My Company want to start marketplace: how to add data for 35k unique products??
My company is a distribution company that handle about 35k products across 20/30 different vendors. We do not produce anything. Now the board want a marketplace/e-commerce with a ton of technical details of our spelled products. These vendors are not so well…
My company is a distribution company that handle about 35k products across 20/30 different vendors. We do not produce anything. Now the board want a marketplace/e-commerce with a ton of technical details of our spelled products. These vendors are not so well developed on websites/e-commerce and the best source of information for all of them are PDFs of their products catalogues, we all the technical data resides. Now, im intrigue in how we could load all of the data into our future system and how we should maintain it (avoid staling information and products revisions) How do Companies manage to load into a PIM that large amount of data entry? I mean, in our business fields products have a lot of (from 6 to 20) different attributes/dimensions, I cannot image 10 people working months just to add data. There are some smart alternatives to pure human data entry ?
Collected discussion
This is less an ecommerce question and more a data ingestion / PIM / ETL problem. You’ll probably get better answers in r/dataengineering or r/ETL. What you’re really dealing with is structured extraction from unstructured vendor PDFs and building a pipeline to normalize and maintain product data at scale.
Thank you ! I will ask there also!
No we are not, we manage all the task (logistics and warehouse) internally. Now we manage all sku in our ERP system but having a PIM is a different kind of matter. In our ERP Rachel sku have fewer fields to manage (ex price, description, category etc) than a product in a PIM
Great - data guys will be able to help your more. In regards to marketplace - what is your central point to manage all different marketplace publishing
Your comment in r/ecommerce was automatically removed as your 'post' karma is below 10 (we do not consider your total karma; your post and comment karma are separate numbers and must both meet their minimum requirement). You can increase your post karma by posting in other areas of Reddit to earn upvotes. The higher quality the content, the higher your karma will become. I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
Your comment in r/ecommerce was automatically removed as your 'post' karma is below 10 (we do not consider your total karma; your post and comment karma are separate numbers and must both meet their minimum requirement). You can increase your post karma by posting in other areas of Reddit to earn upvotes. The higher quality the content, the higher your karma will become. I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
Your comment in r/ecommerce was automatically removed as your account is too new (accounts must be at least 30 days old). Try again a little later. I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
Your company being a distributor- how do you mange the 35k Sku inventory currently for all that products? Are you already a 3PL company ?
What do you mean “want A marketplace”? Do you mean make a marketplace or list on a marketplace?
No one is doing that manually at scale, that would take forever and still be messy. The way this is usually handled is a mix of automation plus cleanup, not pure data entry. First step is extracting data from those PDFs. Tools can pull structured data out, but it won’t be perfect, so you treat it as a first pass, not final data. Then you normalize everything. Different vendors will describe the same thing differently, so you need a standard format for attributes across all products before importing into a PIM. After that, you import in bulk and only fix exceptions manually instead of typing everything from scratch. For maintenance, the key is not repeating this process. You need a way to get updates regularly from vendors, even if it’s simple spreadsheets or periodic data dumps, otherwise it will go stale again. So it’s less about adding 35k products once and more about building a system where data flows in and gets cleaned, not created manually each time.
The question is, how do these vendors supply the data. Is it in an industry standard format like auto parts? Random PDF catalogs? Csvs?
You're going to have to peice it together from a few different sources. But first, find out what format your e-commerce site can injest best. You'd hate to spend 3 months getting everything into JSON when it wants CSV. Think of: Reaching out to the vendors for spreadsheets of product attributes. Scraping their websites. Maybe the images are reliably at https://company/parts/partNum.jpg You don't even need to scrape if the product details are loaded through an AJAX call...public API! Demo a few PDF to data programs See how LLMs can help you with the repetitive bits
We are building a solution for cross listings. And bulk import by barcode UPC or even photo, and automatic categorize them. Maybe we can help you out to sort these mess.
I wouldn't try to import all 35k products in one big push. Start with one category first and clean the structure there: product names, attributes, specs, images, categories, filters, and what buyers actually search for.
We have a few agentic flows for data extraction using LLMs and Web. I may help with this problem.
This comment was deleted.
This comment was deleted.
This comment was deleted.