What If Human Motion Became Software?

I keep coming back to one question: What if human movement could become software?

I have been exploring computer vision, cloning technologies and simulation for a while, and these ideas keep connecting in my head. Voice can already be represented digitally. Appearance can be reconstructed. Language models can reproduce parts of how someone communicates. Video models are getting better at understanding the visual world.

So naturally I started thinking about another part of us: movement.

Not simply recording a person walking. I mean extracting enough information from ordinary video that another machine could understand how someone moves, interacts with the world and performs a physical task. Then perhaps reproduce parts of that behavior inside simulation or eventually inside a robot.

The rough idea looks like this:

video → 3D human → motion representation → robot retargeting → physics → control → real robot

Written in one line, it sounds simple. In reality, almost every step is difficult.

A Video Is Still Just Pixels

When I watch someone pick up a cup, I don't think about pixels. I see the person, the hand, the cup and the action. My brain already understands the scene.

A computer starts somewhere much more primitive. A video is basically a sequence of images, and every image is a grid of numerical color values.

From those pixels, the machine has to work out where the person is, where the hands are, how the body is moving, how far away objects are, whether the camera itself is moving and what is actually happening in the scene.

Then it has to answer harder questions. Did the hand actually touch the cup? When did contact happen? Was the foot planted on the ground? What was the person trying to do? How much force was involved? Which movement was necessary for the task, and which part was simply the person's individual way of moving?

This is where the problem becomes much more interesting than pose estimation.

Turning Pixels Into a Human in 3D

The first step is becoming increasingly possible. Computer vision systems can already take ordinary video and estimate the human body in 3D.

Instead of seeing millions of unrelated pixels, we can begin extracting something more structured: body position, joint rotations, hand trajectories, foot positions, orientation, velocity and movement over time.

That is already a huge transformation. We are moving from what the camera saw to a structured representation of what the human body did.

But even a perfect reconstruction of the human is still not enough.

A Human and a Robot Are Not the Same Body

Suppose I perfectly reconstruct somebody walking. It might seem that I could simply copy their joint angles onto a humanoid robot.

That does not work.

A human body has muscles, tendons, soft tissue, flexible joints and extremely complex hands. A robot has motors, rigid links, encoders, torque limits and mechanical constraints. The proportions are different. The mass distribution is different. The joint ranges are different. The feet are different. The balance is different.

So if the human knee bends by a certain amount, that does not mean the robot should copy the exact same angle.

What we really want to preserve is something more abstract: keep this foot planted, move the hand along this path, turn the torso this way, maintain this rhythm, reach this object and finish in this position.

Then the robot has to figure out how to satisfy those constraints using its own body. That is part of what motion retargeting tries to solve.

And even after retargeting, there is another problem: physics.

Animation Is Easy Until Gravity Arrives

A digital character can perform almost any movement on a screen. It does not necessarily have to obey the real world.

A robot does.

The moment we take a beautiful reconstructed motion and put it into a physics simulator, gravity arrives. Then friction, momentum, motor limits, sensor noise and differences in the floor arrive too.

Then the robot falls.

This is one reason simulation interests me so much. Simulation is not just a place to make a robot look realistic. It is a laboratory between information and reality.

You can reconstruct a person moving, place a humanoid in simulation, try to reproduce the motion, watch it fail, understand why and improve the system.

Maybe the feet slide. Maybe the center of mass moves too far. Maybe the required joint movement is impossible. Maybe the robot works on one surface and falls when friction changes slightly.

Every failure tells us something.

Motion Is Not the Same as Skill

Now imagine something more complicated than walking.

Imagine a mechanic who has repaired the same machine for thirty years. You can record a video of the mechanic. You can recover their body motion. But the skeleton is not where most of the valuable knowledge exists.

The real knowledge might be in what they notice first, which sound tells them something is wrong, how tightly they hold a part, when they reduce pressure, which hand stabilizes the component, where they look and what they do when the expected result does not happen.

A physical skill combines perception, memory, decisions and movement.

So the real pipeline may eventually look more like:

pixels → 3D world → human motion → objects → contacts → task → physical constraints → machine action

That is much harder than simply drawing a skeleton over a person. But it is also much more valuable.

What Makes Someone's Movement Their Own?

This is another part of the problem that fascinates me.

Imagine two people walking from the same door to the same chair. The task is identical, but they probably do not move in exactly the same way.

One person may have faster steps. Another may swing their arms more. Someone may lean forward slightly. Someone may turn their whole torso before changing direction. Someone may have a tiny asymmetry in their walk.

If you know a person well enough, sometimes you can recognize them from their movement before you even see their face.

That starts to look like a kind of motion signature.

Not one permanent fingerprint, because our movement changes with age, mood, injury, tiredness and environment. But perhaps a set of characteristics that appears repeatedly.

Then we can start separating two things: what the person is doing and how that person tends to do it.

Instead of storing only a video of somebody walking, perhaps one day we could represent something closer to:

action = walking
destination = kitchen
speed = normal
style = person_A

Then an avatar or robot could generate a movement the person was never recorded performing while still preserving some of their characteristic physical behavior.

That, to me, is much more interesting than simple motion capture.

From Motion Capture to Physical Identity

This makes me think about a larger question.

Today we are already learning how to preserve different parts of a person digitally. We can reproduce voice. We can generate someone's appearance. Language models can imitate parts of communication style. Memory systems can store years of information. Digital avatars can talk.

But humans are not only language and appearance. We also exist physically. We communicate through posture, timing, facial movement, gestures, reactions and the way we interact with the environment.

So imagine a future digital representation containing different layers: memory, knowledge, voice, appearance, personality, relationships, physical behavior and skills.

Some parts could live inside an avatar. Some could exist inside simulation. Some could eventually control physical machines.

This does not mean we have copied a human consciousness. That distinction is important. It means we are gradually learning how to preserve richer representations of observable human behavior.

Why Pantheon Comes to Mind

One reason I found Pantheon so interesting is that it takes this idea to an extreme.

In the series, a human mind can be scanned and turned into an Uploaded Intelligence. That is science fiction. We cannot do that today.

But the question behind it is fascinating: What exactly would have to be preserved for a digital entity to still represent a person?

Memories, personality, voice, language, relationships, movement, skills, neural structure? Maybe all of them. Maybe there is something fundamental we still do not understand.

The interesting part for me is not predicting whether Pantheon becomes real. It is thinking about the smaller technologies that might slowly move in that direction.

What About Neuralink?

Brain-computer interfaces create another possible path.

But there is an important difference between what exists today and what science fiction imagines. Neuralink cannot currently scan and copy a human brain. Its current technology records limited neural activity and tries to decode things such as movement intention so a person can control an external device.

That is already remarkable, but it is nowhere near copying all the neurons, connections, memories and internal states of a human brain.

Still, imagine the future if several technologies develop together.

Video could capture observable behavior. Audio could capture voice. Language systems could preserve knowledge and communication. Personal data could preserve memories and history. Wearables could capture parts of physiology. Brain interfaces might eventually capture information that cannot be observed externally. Simulation could provide a digital body and world. Robotics could provide physical embodiment.

Then the question becomes much bigger:

How much of a human can eventually be represented computationally?

I do not know the answer, but I think it is a fascinating question.

Even a Perfect Brain Copy Would Still Have a Problem

There is something else that often gets skipped.

Imagine that tomorrow someone somehow created a perfect digital copy of a human brain. That still would not automatically create a robot that behaves exactly like that person.

A human brain learned inside a human body. It learned balance through human feet. It learned to grab things using human hands. It learned gravity through years of falling, walking, running and interacting with the world.

Now put that intelligence inside a robot with metal feet, electric motors, cameras, different proportions and different sensory delays.

How does it move?

Knowing "I want to pick up that glass" is not the same thing as knowing exactly how every robot motor should move while maintaining balance.

There is still a translation layer between intention and physical action.

This may be one of the deepest problems behind Physical AI:

How does intelligence learn to inhabit a body?

Maybe future robots learn from human demonstrations. Maybe from simulation. Maybe from neural signals. Maybe from world models. Probably from some combination of all of them.

The Industrial Version May Arrive Much Earlier

The first major impact of this idea may not be digital immortality.

It may simply be preserving human expertise.

Imagine again the technician who has repaired the same machine for decades. Today we might ask them to write instructions before they retire, but many things they know are difficult to write down.

They notice a vibration. They hear a slightly different sound. They feel resistance. Their hand changes position automatically. Their body knows what to do before they consciously explain it.

A huge amount of human knowledge exists like this. It was never written. It exists in behavior.

If machines could observe enough of those interactions and transform some of them into structured physical information, we could preserve parts of expertise that currently disappear.

Not the human or their consciousness, but something valuable that person learned about acting in the physical world.

That alone could be extremely important.

The Missing Layer

We have spent years turning different parts of the world into information machines can process.

Text became tokens. Audio became digital signals and embeddings. Images became representations.

Now I think one of the fascinating missing layers is:

human activity → machine action

To build that bridge we need computer vision, 3D reconstruction, object understanding, robotics, biomechanics, physics, simulation, control and machine learning. We also need enormous amounts of feedback from the physical world.

It is a much harder problem than building another application around a language model.

That is exactly why I find it interesting.

Starting Small

I don't want to begin with:

How do we digitize a human?

That question is too big to be useful technically.

I would rather begin with something much smaller:

Can I take an ordinary video of a human and make a simulated humanoid reproduce the movement?

Then ask the next question.

Can I recover reliable 3D motion? Can I separate human movement from camera movement? Can I detect when the feet touch the ground? Can I understand interaction with objects? Can I retarget the motion to a completely different body? Can I keep the robot balanced?

Can I preserve the important part of the movement instead of copying every joint? Can I extract something about the person's movement style? Can that style generalize to an action they were never recorded performing? Can the representation work across different robot bodies? Can the same behavior eventually survive outside simulation?

Each answer reveals the next problem.

Maybe that is the path. Not human → digital human in one impossible jump, but:

pixels → structure → motion → behavior → physics → simulation → action

Then eventually:

action → physical intelligence

I don't know whether the future looks like Pantheon, humanoid robots, brain interfaces, digital avatars or something completely different. Most likely several of these paths develop together.

What keeps pulling me toward this problem is much simpler.

A huge amount of human knowledge has never been written down. It exists in how we move, react, touch, manipulate, adapt and interact with the physical world.

If machines are going to become truly useful in that world, they will eventually need a way to understand that knowledge too.

And perhaps somewhere along that path, parts of what make each human physically unique will become something machines can understand, preserve and carry forward.