One approach to enabling robots is the video training approach for AI/Neural Net systems. In the case of self driving cars, this amounts to '1 million hours' of training video fed into the training supercluster and then outputting the resulting parameters to the inference engines.
If instead we're training a robot to do the laundry, I see the entire chore in it's full activity:
- Roam the house looking for the kid's socks
- Gather the dirty clothes from baskets & hampers
- Sort the laundry into hot/warm/cold loads, per the desires of the owner
- Load the washer
- Add detergent. liquid or capsule
- Set the cycle parameters, per make/model of washer and owner-desired settings
- Start the load
- Promote the load to the dryer when finished.
- Set the dryer cycle parameters, per make/model and owner-desired settings
- Remove clothes when dryer completes
- Empty lint screen
- fold clothes and sort into owner
- Replace clothes, hang on hanger, place into drawer, based on each owner's desired placements
Then, the fun begins...
- Handle exceptions like, load out of balance, power outage interrupt, washer starts leaking, ...
My question, (thanks for reading through all that!), do you think it would require more, less or the same amount of training video as self driving cars to enable this complete approach to 'doing the laundry', so that when a robot owner purchases the 'do the laundry' ability, the parameters can be downloaded to their robot and then never have to care about the laundry again?
You are getting at is what folks are calling a robotics 'foundation model' and the comparison is usually closer to something like ChatGPT (a general purpose model that knows a little about a lot of things) rather than self-driving (does one thing very very well). I think the fact that the stakes are lower for humanoids (they generally don't weigh thousands of pounds and travel at 80mph) means that you might need fewer 9s of reliability, and I think data needed probably scale superlinearly with the number of 9s you want, so we have a much larger and varied domain (all home tasks) but a lower success threshold. I'm not sure. The other big bet being made by folks like Figure and Tesla is combining first person video of humans doing stuff with robot video and using it as a data multiplier, which seems to be working to some extent, which is pretty exciting if data is your bottleneck.
I believe this old video from Boston Dynamics satisfies gold for door opening, unless you're being particularly picky about "humanoid" or "ML required" requirements.
Did you “borrow” my sock Turing test or did you come up with it in parallel? No worries just curious. I’ve been using that example for a quite a well. It’s a great test!
I think after the age of large-scale imitation learning, most tasks could be overfitted with enough high quality human demonstration data. Setting tasks as benchmark may not be the best way to measure the performance of generalist robots.
Want to hear your thoughts on next step for evaluations.
Still waiting for Event 6: Whole body manipulation
Bronze: Remove fitted bedsheets (assume mattress is heavier than bedsheet tear strength)
Silver: Put fitted bedsheets on
Platinum: Put duvet cover on a comforter (opportunity for multi agent)
Hi Benjie, I hope you've been great!
One approach to enabling robots is the video training approach for AI/Neural Net systems. In the case of self driving cars, this amounts to '1 million hours' of training video fed into the training supercluster and then outputting the resulting parameters to the inference engines.
If instead we're training a robot to do the laundry, I see the entire chore in it's full activity:
- Roam the house looking for the kid's socks
- Gather the dirty clothes from baskets & hampers
- Sort the laundry into hot/warm/cold loads, per the desires of the owner
- Load the washer
- Add detergent. liquid or capsule
- Set the cycle parameters, per make/model of washer and owner-desired settings
- Start the load
- Promote the load to the dryer when finished.
- Set the dryer cycle parameters, per make/model and owner-desired settings
- Remove clothes when dryer completes
- Empty lint screen
- fold clothes and sort into owner
- Replace clothes, hang on hanger, place into drawer, based on each owner's desired placements
Then, the fun begins...
- Handle exceptions like, load out of balance, power outage interrupt, washer starts leaking, ...
My question, (thanks for reading through all that!), do you think it would require more, less or the same amount of training video as self driving cars to enable this complete approach to 'doing the laundry', so that when a robot owner purchases the 'do the laundry' ability, the parameters can be downloaded to their robot and then never have to care about the laundry again?
Hey Bill! Good to hear from you.
You are getting at is what folks are calling a robotics 'foundation model' and the comparison is usually closer to something like ChatGPT (a general purpose model that knows a little about a lot of things) rather than self-driving (does one thing very very well). I think the fact that the stakes are lower for humanoids (they generally don't weigh thousands of pounds and travel at 80mph) means that you might need fewer 9s of reliability, and I think data needed probably scale superlinearly with the number of 9s you want, so we have a much larger and varied domain (all home tasks) but a lower success threshold. I'm not sure. The other big bet being made by folks like Figure and Tesla is combining first person video of humans doing stuff with robot video and using it as a data multiplier, which seems to be working to some extent, which is pretty exciting if data is your bottleneck.
I believe this old video from Boston Dynamics satisfies gold for door opening, unless you're being particularly picky about "humanoid" or "ML required" requirements.
https://youtu.be/aWV9sIE6stU
Did you “borrow” my sock Turing test or did you come up with it in parallel? No worries just curious. I’ve been using that example for a quite a well. It’s a great test!
I think it is a case of parallel evolution. I was thinking of all the types of laundry folding you'd want to do other than t-shirt folding... :)
Exactly! I’m sorta obsessed now as I go through my daily tasks analyzing how a robot might handle it!
great bengi
Nice blog Benjie! How do you think of recent submission by PI? https://www.pi.website/blog/olympics
I think after the age of large-scale imitation learning, most tasks could be overfitted with enough high quality human demonstration data. Setting tasks as benchmark may not be the best way to measure the performance of generalist robots.
Want to hear your thoughts on next step for evaluations.
Super interesting. Much more to add, of course, eventually. Talked about this here: https://www.forbes.com/sites/johnkoetsier/2025/12/24/the-robot-olympics-will-have-zero-sports-heres-why/
The round knob handle is a very bad design and it's good the civilisation is dropping it.
It would not be easy to come up with a more adversarial design.
Basically it works as intended only for a healthy able-bodied human adult in their prime not carrying anything whose hand is unsoiled.
Bonus festure: unparalled germ spreading ability.
👨🍳👄
Could we adjust this challenge so there is no reason for robotics labs to have these evil devices?