Writing a Game Engine in 2017

Writing a game engine in 2017, what does that look like? Should a developer download Unity? Gamemaker? Love2D? Almost certainly, correct? Why would anyone want to create a game engine… In 2017? This will be my last post for some years, so lets see if I can make it a decent one.

There are only a couple reasons to create a custom piece of technology, game engines included:

  • It is personally fun to the creator
  • Some kind of advantage can be utilized

Really these can be coalesced into a single piece as having fun while doing the hard-work of making a game is huge advantage. The types of advantages that come along with a piece of technology are:

  • Specialization
  • Performance
  • Breakthroughs/Innovations
  • Control

Again these can probably all be coalesced into the Breakthroughs/Innovations category. Custom tech should exist to give the creator an advantage somehow, and the types of advantages that really matter in terms of product success are the innovations and breakthroughs.

Usually the breakthroughs are in some kind of new graphics technique, new kind of AI, or new utilization of hardware. However, breakthroughs can be made in any area of game development, not just the vertical scaling of certain common aspects. A breakthrough in expression, for example, can yield tremendous success. Content iteration time, highly efficient or specialized tools are a couple other examples of great places to gain a competitive advantage. Here is a long reddit post I wrote on this topic, please do read it.

One last thing is that making a custom game engine will require quite a bit of expertise, and in order to write a good one will require a lot of experience. It’s not necessary to make a good game engine to make a good product, but it certainly helps! At the very least writing a game engine and finishing a product (finishing a game) with that engine will immediately make an engineer go from no-hire to immediately-hire. That said I’ll link to a document I was commissioned to write that covers a bunch of game engine and programming related topics, so any newer programmers can take a look and use it for a reference.

Runtime Compiled C++

I would like to share one kind of breakthrough more or less popularized by Casey Muratori from his Handmade Hero YouTube series. The idea is to use C/C++ for everything. Personally I wouldn’t dare use pure C because function/operator overloading is a very good feature. Then place game implementation, all of it, into a DLL. The entry point holds memory allocation features, and manages the game DLL. At run-time the game DLL can be hotloaded and swapped out whenever the user likes. C++ becomes as if it were a scripting language.

To make this work a lot of care needs to be placed on making sure compile-times stay very low. Here is a thread where I talked a bit about compile times for larger projects. Casey Muratori has released a ctime utility for profiling compile terms. Perfect :)

Finally, here is an old post of mine talking about some pros and cons of this style. Please do read this post. It contains majority of pros and cons, and talks about using code “as an editor”, as in, a general purpose editor (like level editor, or sequence editor, or cut-scene editor).

And last I have a working example for Windows here on Github. In hindsight I would probably recommend implementing this with SDL. Since the entrypoint for this style of game engine does not need to be recompiled at run-time, any type of heavy-weight libraries that can be unknown to the game implementation can safely hide in the entrypoint without affecting compile times much. SDL has some tools for dealing with loading/unloading dynamic libraries that can be used for cross-platform development.

Game Objects/Entities in Runtime C++

The biggest limitation for most programmers would be the total destruction of all function pointers, including the pointers to vtables that most compilers use to support the C++ virtual keyword. Since most compilers will likely implement vtables by storing a pointer in C++ objects to some kind of static table of function pointers, reloading dynamic libraries can move the location of vtables. This can leave dangling pointers in leftover C++ objects.

This can be solved with complicated C++ serialization techniques and reflection.

Or, this can be ignored completely.

Back in the days of GBA some games would want to make use of polymorphism (I recommend clicking the link, it is not Wikipedia). Polymorphism is an incredibly useful concept, and the typical way to implement polymorphism in C++ is with the virtual keyword. In C function pointers are used. With runtime C++ neither of these is an option (or so it seems).

When a dynamic library is unloaded and reloaded all functions are assigned new locations in memory. The nice thing is that the static variables and static file-scope bits of data are all cleared to zero if unitialized, and initialized if they have initializers. This is nice since function pointers in these data sections will be properly initialized to new function addresses upon load!

By taking a page out of old GBA C-implementation techniques we can reimplement polymorphism in a very straightforward way.

The idea was to have entities contain a virtual table (or vtable) of function pointers. However, there is no good reason to duplicate virtual tables into every single entity in the game, and instead this data really should be made static. Each entity can contain an integer identifier to describe what kind of entity it is. This ID can be used to index into a static array of vtables.

The VTABLE

Here is how I define what my entities look sort of like in my own personal code:

Anything that is universal to all entities no matter what can go into the Entity struct. For now I just have double linked list pointers. I made add later, I may not add more later. It is unimportant. The important part is the id.

The id is used to index into a table that looks like this:

Every entity can Update or Draw itself in a polymorphic way. Making any entity update or draw itself is a matter of:

Personally I didn’t bother with type-safety. However some type-safety can be added if anyone cares enough.

Instead of each entity holding a pointer to a vtable like the C++ compilers do, they just hold an integer that indexes into a global array. The global array is properly initialized upon dynamic library reloading. It all works perfectly.

Some years ago in university I actually wrote out this kind of vtable stuff for a school club, as an example to students in my following year. Turns out it was incredibly useful for modern day game engine implementation. Feel free to take a peek if an example is of interest (albeit an older example).

Run-time Compiled C++ is also a good option! This github repo implements a much more full-featured style than the style this article describes. At the very least this code serves as a really cool learning resource. I have peered over this github repository quite a few times in the past years.

Smaller Compile Times

In order for this style to work compile times must be kept to a minimum. To facilitate this I have implemented a slew of useful libraries called tinyheaders. Feel free to use them in your own game engine. They accomplish tasks like multiplayer netcode, playing/looping sounds, collision detection, and a bunch of other odds and ends. Writing code like shown in tinyheaders, or on HandmadeHero is important to keep compile times at bay.

Truth be told almost an entire game engine can be constructed from these tinyheaders alone!

Unfortunately for many, this rules out the use of most C++ libraries, especially the ones making judicious use of templates. Libraries like Boost, as the prime example, will not work in this case.

Another important trick is the unity build. I have heard rumors of ridiculous claims, that some game engines when compiled as a unity build gain a 10% boost in performance. I have heard that unity builds are a more natural compilation scheme for C/C++. In my experience using a unity build makes compilation faster, if the entire project is written with compile times in mind. Compilation performant code should be aspired to from project inception! Keeping compile times under a few seconds is very important for iteration times.

Hotloading Assets, not just Code

Packaging up and creating assets is the other part of “compile times” that should be handled with care. For example it is quite tempting to place texture atlas compilation into the code compilation step. This can pretty quickly degenerate compile times in an unnecessary way. Instead some kind of alternative should be devised.

In my personal code I have a button to hotload assets, and upon game open (in debug mode) asset packaging is invoked. Since run-time C++ hotloading is used I actually try, for fun, to not close the game all day as I develop! So initial opening of the game does not happen very often, so it is OK is asset compilation takes a little time.

A nice commenter over in my Texture Atlas post pointed out another good solution; there is a program running that scans over asset directories looking for work to do (like atlas compiling). Whenever it sees work to be done, it kicks off and does it. The game itself can be notified of new assets (either by scanning file timestamps itself), or through some other means — whatever is preferred. TCP or UDP are some examples for inter-process communication. The asset scanner can also just exist within the game itself as a separate thread, sleeping whenever necessary.

Always On-Line Philosophy

Suddenly it starts making sense to just always leave the game running. Code can be hotloaded, and assets can be hotloaded. There should only be a need to close and re-open the game if data layouts change (see the alternatives listed above if you want to support data layout changes see the above alternative link).

The nice thing about the code-hotloading is the way it encourages the programmer to use code “as an editor”. Suddenly all the work of making some data driven is all about writing code as if it were the raw data! Incredible.

Here’s a very typical example. Take some more canonical or traditional styled C++ for a simple player movement class and function:

Okay, this is fine and dandy, it does work. After some playtesting it is realized speed multipliers would be fun. Lets code that:

Great! So in this style of development the game would need to be closed, recompiled, reopened many times to achieve this effect. Often times these games are not written with compilation speed in mind, so maybe compilation takes a good 3 minutes. Loading the game takes a good 15 seconds. Going through some menus takes another 30 seconds. We are now looking at a solid 5-7 minutes of completely useless bullshit that gets in the middle of important problem solving and iteration. Unacceptable! If readers wonder where the extra 2 minutes came from… Experience shows that once a long compilation is finished the engineer will already be watching youtube or browsing reddit, unaware for a solid minute or two (at minimum) before doing the next steps.

Now the designer wants to do some heavy tuning of the multipliers, their stats, duration, when they are triggered, etc.

The “old” way to solve this comes from the conventional wisdom: DATA DRIVE THE THINGS.

So now we spend some time spitting out these multiplier floats into yet another shitty xml file, and the designer can modify this xml file and see changes in-game while the game is live. Woopty doo. So how long did it take to implement this xml stuff? Was a new library integrated into the project to read and save xml files? How long does it take to compile? Or does it add yet another DLL dependency?

Instead of all this garbage, how about we treat the C++ code as data itself. Instead the player thingy can look like:

The speed can be tuned by anyone at any time that has a little bit of C knowledge, especially while the game is running! Brilliant.

But what about multipliers? Assuming a data layout change, and some kind of queue is created to hold multipliers, the game will still need to close and re-open. But! Just once, and we’ve been optimizing our compile times so it’s really not a big deal.

Multiplier support:

Alright, great! The internal queue system applies bits to the m_flags as necessary, and unsets them. This style of development really hammers home the differences between run-time RAM data, and on-disk data (code, assets). The above snippet places disk data into constants in the code, and the run-time mutable RAM is in the m_pos, m_direction and m_flags pieces.

Any code constant can be modified live and instantaneously iterated. Amazing.

But Randy (the idealized naive viewer will say) isn’t that just hacky hard-coded code??? Yes, it is hard-coded. Hacky? Sure, whatever. Label it “hacky”. But the facts still remain: this style of development has crazy good iteration time. Obviously it is requiring designers to have solid C understanding. This makes this style out of reach for anyone that is not good at C. This can be viewed as a downside. This can also be argued as a very good plus-side. To each their own!

The Code is the Editor

The point is the style of code can be shifted. Code becomes an editor, a live editor. Instead of spending time creating UI based dev-tools code itself is the dev-tool. Animation layout and definitions can be placed into structs forming data tables. If initializes at file-scope, these can be hotloaded and tweaked at run-time.

Imagine using a very cool tool like Spine (video link). The video shows the user doing some fundamental operations: attaching bones together, defining animation curves, keyframes, and time deltas. All of this can be done in C. If a programmer is comfortable in C all of these pieces can just be placed directly into code!

What is the fundamental movements of the mouse and keyboard while using a tool like Spine?

  1. The mouse moves
  2. Some things are drag and dropped
  3. Some keys are typed to define numbers or names

How does this relate to C code?

  1.  The mouse moves to another line of code
  2.  Some things are copy pasted from one spot to another
  3.  Some keys are typed to define numbers or names

HOLY MOLY IT IS THE SAME

As long as the C code is well-written and the system is designed well, it can behave very closely to a very good editor like Spine. But! The iteration time is *instantaneous* as far as getting these things in game and on game-screen goes.

It seems like a zero-cost benefit, but there is a real tradeoff here. The designer must be good at C. The animation editor must be good at C. Everyone must be very good at C. That’s a huge baseline! Being good at C is really hard, and so this option is not for everybody.

ECS is Garbage

Components are garbage. Entity component systems are garbage. OOP is garbage. All acronyms are garbage. Just write the god damn game. Solve specific problems with specific code solutions. Refactor the code as necessary. Everyone that bothers writing “an ECS system” is either still learning core fundamentals, or just wasting their time.

This post describes my personal plan of action towards creating a custom game engine, and the focus has been entirely on simplicity and iteration time. Please, readers, do not focus on “run-time object models”, “entity component systems”, or any other garbage on the internet. Truth is all of these people have never shipped a good game, or made a good product. Just go look at Love2D source, or GameMaker, or whatever have you. These products are successful, and they don’t bother around with acronyms. They just solve problems for their customers.

Your game, and your game engine should be solving *real* problems. Actual problems with clear definitions. Problems that you have seen. The game engine should be solving problems regarding iteration times, innovation, or specialization. The game itself should also be solving some kind of problem. What does the customer need? What do they starve for? How does the product relate to these needs and desires? Often times talk of ECS is pure hype.

The way I see it ECS is an attempt to construct a methodology for software engineering. Software engineering being more about API design, organization, code longevity etc. compared to just raw coding. Software engineering is a difficult and unnatural skill that more or less requires experience. Writing a good game engine will take experience, not an ECS/OOP/DOD/InsertDumbThingsHere acronym. Trial and error, sweat and blood, these are how good software is written, game engines included.

It seems some others are starting to speak up against ECS, here’s a thread from gamedev.net talking about this.

Addendum: The above section is just trying to attack acronyms and naive advice. As seen below (in the comments) there are some games that were released which tried to implement some kind of ECS or component system, and successfully shipped. Sure! But the exception makes the rule, right? Of course a team of solid engineers can create and ship a game with some kind of ECS, but still, it took a team of solid engineers to do so. Just take the above section with a dose of skepticism.

Know what you Want

Since the primary advantages of custom tech involve innovation, and making something truly special, it becomes paramount to know what you want. What do you want out of your product? Are you making a game, or a game engine? If you are not making a game, then why make a game engine? That’s ridiculous. What kind of product is one without customers?

Know what kind of game you want to make. If this is unknown then why are you reading a post about making a game engine? Please just go play with Unity until you know what you want. Then we can play ball when you get back.

The Long Grind

Making a custom piece of technology is a long grind. Making a good game engine in order to produce a quality product is a long grind. There will be little to no external source of motivation. Either you have the guts to just do the hard work, or you don’t. Making something great requires perspiration and knowledge, and a little opportunity. If you’re able to read this blog post you have enough opportunity.

Figure out what your personal strengths are and how you are as a person. Play to your strengths. Don’t try to build up weak spots and make them your strengths, unless those weak spots are who you truly are. For many people this precludes writing a custom engine, but that’s fine! Making a custom engine is just one option, and should serve the creator (not vice versa).

Farewell

This may be the last blog post I ever write on this website, and at the very least will be the last one for some years. Hopefully it is useful to someone. Please leave comments or suggestions and I’ll edit this post and beef it up over time.

Cheers.

Extra Arguments

A lot of readers dislike the above section that attacks ECS as an acronym and concept. For the sake of posterity, and for anyone interested, I’ll respond to some common pro-ECS arguments seen on the internet. Feel free to skip this section otherwise.

I will reference this tidbit from an old reddit thread and respond to each point. The points raised in the thread represent very common arguments on the internet in favor of ECS. The goal here is to provide curious readers a voice that opposes popular opinions, that way readers can hear more than one side, and subsequently form their own opinion.

ECS enables composition which is one way to solve the problem of sharing behavior. Inheritance solves the same problem but results in inflexible hierarchies. See the bugs in League of Legends caused by “everything extends Minion.” ECS lets you pick and choose any set of qualities for your entities without running the risk of bringing in qualities you don’t want and without duplicating code.

The idea here is “use ECS because composition is good”. I’m sure most readers are familiar with the old is-A vs has-A stuff. Most readers have heard about features floating up inheritance hierarchies. However, ECS or any other fancy acronyms are not needed to implement some kind of composition. Simply defining some POD structs and including them inside of other structs can implement the concept of composition.

The gripe I have is when  we say “do this ECS thingy to get composition benefits”. This is a methodology, a cookie cutter solution, i.e. no critical thinking involved. It sounds like that dumb bag of tricks metaphor everyone spouts about angrily on forums. Bad engineers rely on their bag of tricks to solve problems without thinking. Bad engineers implement acronyms simply to get “benefits”. Bad engineers waste time making up idealized problems only to spend energy solving them, usually for the sole purpose of measuring e-peen.

ECS provides a path to lock-free parallelized processing of your model. You know ahead of time what components each system reads from and writes to. With that knowledge alone, you can automate the parallelization of your systems. This helps solve the problem of finishing processing in under 16ms.

Depends on the definition of ECS, which constantly changes and nobody really seems to be an authority on the subject. Lock free algorithms, or good multi-threaded algorithms in general can be implemented without ECS. Multi-threading requires separation of data and problems, neither of which are exclusive to ECS. Anyone can implement a game without knowledge of ECS and include threads in an efficient manner.

Additionally the part about “automate the parallelization of systems” part is very naive. I assume this is trying to talk about setting up some kind of dependency graph, and some sort of task system can be used to do work on a threadpool. Okay, that’s fine and dandy, but it isn’t necessarily a good thing. Dependency graphs and parallelization come with tradeoffs. For example code flow becomes completely lost in some data structure that represents the dependency graph. Suddenly a programmer cannot easily trace a callstack, or understand the flow of how state mutates over time without referring to dependency graph. This is a really big abstraction that comes with a heft abstraction cost.

I have never seen a single discussion of ECS that has actually shipped a game that talks about the tradeoffs involved here, and how to gauge tradeoffs against a certain game project.

But of course, assuming random people on the internet have actually shipped a product with an ECS is a ridiculous assumption; pretty much all articles I have personally seen on the internet regarding ECS’s were definitely not written by anyone with real experience.

Use an ECS because:

  1. Cache misses incurred by the more common architectures
  2. Ensuring composability, simultaneously erasing bogus dependencies between code and data

For point one: why not just solve the cache miss problem directly? If a particular game is actually having cache miss related performance problems, why not address that problem instead of relying on some goofy ECS-method to solve it for you?

Saying “solve cache misses” is just ridiculous. What cache misses? Where is this cache miss code and how is it specifically a problem? Oh wait, these cache misses must be yet another idealized problem born of e-peen measuring instead of actual experience or actual problems in an actual shipped game.

Any half-decent programmer will know how to avoid a cache miss. ECS is not needed to learn about caches.

Point two sounds interesting, but I just don’t know what it is saying. It sounds something like the point Erin from the comments made (should code belong in A or B), which I responded to elsewhere in this post.

Rigid classes rarely match game objects. Making them flexible can lead down paths of optional pointers (dynamic), or multiple inheritance (static), or just building massive “everything” objects. I prefer flexible and sparse representation of game objects (entities) defined by their properties.

This sounds nice I suppose, but I just don’t know exactly what half of these terms are. What is “rigid class”, what is “optional pointer”? I just cannot respond to this without delving into definitions of all these terms.

Updating by “object” easily has issues like entity B (during its update) accessing state of entity A (at frame n+1) and entity C (at frame n).

So? What is the problem here? For the sake of argument I will assume this point is trying to describe a point made by Erin in the comments, something along the lines of “where does this code belong, in A or B?”. This can be a pretty annoying problem and lead to a lot of jumping back and forth across pointer indirection. The jumping back and forth is a problem since it destroys code flow (the ability for an engineer to quickly grok the program’s path of execution).

This can be solved by separating the marriage of code and data, in effect splitting up the concept of “object” into data + code. Code operates on some memory. Code that operates on data A, and code that operates on data B can easily be placed into two categories. Data A and data B are similarly two different conceptual categories.

That’s fine. This can be done without the ECS acronym. These concepts are not ECS specific. A good engineer will not rely on following steps 1-10 of How to Create Your Very Own ECS in order to solve mangled code flow problems.

Lacking uniform representation means it’s hard to add pan-object features. One way is having everything inherit down to a base object, where you can add such things, but that is horrible. Components make this trivial: entities are just IDs to associate components to. So you can decide a “faction” is something with a name and relations which other things can be associated to. Done. Or if you want a debug-tag, define the component and attach it to things: in data or at runtime! No need to touch any other code or change any structs. Modular composition bliss.

This is describing the simple concept of composition. This is not “ECS” or “component based design”, or any other dumb acronym.

These concepts have existed for many decades.

Entity specification (data), and serialization… often a pain. Components make this a 1:1 correspondence: just load properties on an ID. Templating is easy: an entity instance can inherit from a template, and add overrides. Serialization just stores the overridden (local) components.

Honestly I don’t know what most of these terms really mean. I’m guessing the overall point is 1:1 correspondence of serialization to components is nice and straight-forward. OK sure, that’s a good point. But like all other points, why should anyone care about ECS? Defining some POD-structs and making good use of them is as old as my grandfather. Yes, clearly POD-structs are trivial to serialize, so if data within a program is well organized and sorted, then sure serialization should become much easier.

But this does not really require the use of some silly acronym. Any good engineer will already know this.

What all these points show is that online voices that argue in favor of ECS are simply very excited about following a methodology. The idea is if an engineer follows this acronym, or these set of standards, or these “rules” they will get some good benefits. The problem is this is a fantasy. Good code will never come from a cookbook of steps. It requires critical thinking, adaptation, and experience.

TwitterRedditFacebookShare

28 thoughts on “Writing a Game Engine in 2017

  1. Spammish

    Excellent post. Thank you.

    I don’t mean to pry but why farewell? I’ve (quietly) enjoyed your writing for quite some time. Sorry to see you go.

    Cheers.

    Reply
  2. Erin Catto

    We use ECS in Overwatch’s gameplay layer. It is not perfect, but it solved many problems for us. Previously we used EC and the components did the update. This leads to a lot of spaghetti code.

    The problem is that component updates naturally need to grab other components. Then where does the logic go? In componentA, componentB, etc? Then you get into situations where the update of componentA does some of the update for componentB, etc. This leads to a tangled mess.

    By using the S part of ECS, we now have a clear landing spot for game logic and it clearly specifies where game logic belongs. We are also able to clearly specify the order of the systems in the game loop.

    Thinking about the evolution of game logic, you can start with just entities. This leads to old C-style fat entities where you have to hang everything and the kitchen sink off a single structure. This approach has a lot of simplicity, but can start to become a tangled mess as the suite of behaviors grows. It can also lock you out of emergent combinations of behaviors, like a sword that also talks.

    Game developers try to answer this problem by switching to entity-component systems. I think this is a failed intermediate step for the reasons I gave above. ECS is not dogma, it is just an attempt to add some structure and untangle the mess made by  EC.

    On the flip side, ECS does have problems with time. We often roll back time for a specific entity. Unfortunately, this becomes a headache when the entity is being processed piecemeal through multiple systems.

    Reply
    1. Allen Webster

      Actually, it has been my experience that hanging everything off of a struct leads to the most flexibility for emergent behavior, so long as you minimize the degree to which you rely on the entity ID. If you have:


      struct Entity{
      int id;
      union{
      struct Weapon { int damage; int textures_index; ... };
      struct Talking_Character { int personality_index; int textures_index; ... };
      ...
      };
      };

      Then your assessment of the single struct for an entity is correct. Because then you would have situations in your code like:


      void DoAttack(Entity *weapon, Entity *target){
      if (weapon->id == EntID_Weapon){
      ...
      }
      }

      However if you just say “all entities might be weapons” and drop the entity id completely you get:


      struct Entity{
      int textures_index;
      int damage;
      int personality_index;
      };

      Then the DoAttack code can’t check if something is a weapon by an id, but you can still give it a way to check. For instance:


      void DoAttack(Entity *weapon, Entity *target){
      if (weapon->damage >= 0){
      ...
      }
      }

      Obviously this is a very simplified illustration, but it has been my experience that systems that are structured this way are capable of the hightest degree of emergent behavior because any entity can have any property “no muss no fuss”. Certainly the difficulty gets moved to other times/places by this, but emergence is definitely *not* it’s weakness.

      Just a thought :)

      Reply
      1. Erin Catto

        You are correct Allen. I used the wrong example. With just entities (and no C or S), there are several choices:
        – Inheritance hierarchies (OOP style)
        – Fat entities (C style)

        The OOP style has problems with talking swords due to a rigid behavior structure. I think this is a fairly well understood problem with OOP, so I will not get into details.

        Fat entities are essentially entities with components, except the component concept is not well defined. It could just be a struct with a bunch of pointers to component like things:

        struct Entity
        {
        Weapon* weapon;
        Speech* speech;
        };

        These pointers are optional in some fashion or another. So they are essentially like components and you can have a talking sword. At this point you can have two kinds of update:

        You could have EC style update:
        void Entity::Update(float timeStep)
        {
        if (weapon)
        {
        weapon->Update(timeStep);
        }

        if (speech)
        {
        speech->Update(timeStep);
        }
        }

        void GameLoop(float timeStep)
        {
        for (int i = 0; i Update(timeStep);
        }
        }

        Or you could have an ECS style update:

        void WeaponSystem::Update(Entity* entities, int count, float timeStep)
        {
        for (int i = 0; i < count; ++i)
        {
        if (entities[i].weapon)
        {
        // process weapon logic
        }
        }
        }

        void GameLoop(float timeStep)
        {
        weaponSystem.Update(entities, count, timeStep);
        speechSystem.Update(entities, count, timeStep);
        }

        So what is the problem with the EC style update? With the EC style update, entities interacting with each other have different levels of update. So if two entities are talking to each other, they cannot ensure they are both ready to talk at the same time. ECS tries to solve this problem.

        Reply
        1. Philip

          Hi Erin,

          I understand if you cannot disclose details, but could you elaborate some more and/or make some examples of what ECS looks like in the real world?

          In my mind, I imagine very complex stuff is done at Blizzard :)

          In particular, how do entities communicate with each other? For example what happens when the Physics systems find out there is a collision?

          Is it possible to add / remove components to entities at runtime?

          Stuff like Networking, Physics, or Rendering… are they systems of their own or are they more like “global” services that other systems can access?

          Finally, I would be really curious to know if Diablo 3 is ECS based :)

          By they way, I always enjoy your talks!

          Thanks,
          Philip

          Reply
          1. Erin Catto

            That is the nice thing about ECS, the Entities have no capacity to communicate with each other. Instead the Systems handle interactions. This interaction is on even footing, so there does not need to be a primary Entity.

            We do not add or remove components after an entity has been created. Instead, we use flags to enable/disable behavior.

            If something needs a regular update, it is in a system. Ideally systems do not hold data. However, some systems work in a “retained mode”. The physics engine and the rendering system are retained.

      2. Mathew

        The whole point of an ECS is to be able to do something like this:


        // Inside a 'Battle System' class or something like it
        if(GetComponentOfType().Damage >= 0)
        {
        // Do the damage
        }

        Components are an essential part of a Game Engines structure. You don’t want every game object to contain a ‘Damage’ integer, which is why a component based system is the best approach, this way you can choose which game objects have a Damage component.

        Components are meant to be universal, i.e; you can use them on any game object, and they’ll work.

        The fact that this article reads:

        OOP is garbage

        That alone tells me the publisher truly knows nothing of the core fundamentals of any C style language.

        Reply
        1. wareya

          You don’t *want* a monolithic entity class. The goal of ECS isn’t to make it so that not all entities have a damage field, it’s to make sure that system code isn’t placed inside objects and provide the tools necessary to make nontrivial games like that.

          Entities not having a damage field is a side effect, and the moment you check whether an entity has a particular component, you’ve already done something horribly wrong somewhere, and your game is probably going to have performance problems because your systems are interacting with entities they don’t care about at all.

          Reply
  3. Vladimir Kozlov (ai_enabled)

    That’s exactly the points leading to creation of our custom game engine after we’ve finished a game with Unity (well we still improving it and v2.0 just arrived with an huge expansion/DLC coming soon). It’s client-server and it was really boring to write and support the netcode, properly wire all the server stuff with the client… too many layers between something on the server and its representation on the client. There was no UNET at the time, and now, when I compare UNET with our networking features, our game engine is light years ahead in how easy it is to write the networking code (remote calls with async-await, automatic network replication with support for data hierarchies and really minimal binary diff-packages, subscription on any property (with simple lambda syntax), etc). Even modders are able to write networked code effortlessly. Combined with instant recompiling (with Roslyn) and assets reloading, clear and superior C# 7.0 language, it drives our productivity (which is so important as we’re very small team!) and opens great opportunities for modders.
    Regards!

    Reply
  4. wareya

    Implementing archetypical ECS “correctly”, in such a way that iterating over all instances of a component is fast, and without breaking what ECS “promises” in terms of encapsulation and state dependency validity assurance, takes a very long time and requires you to be a very, very skilled programmer, both in the abstract computer science sense and in terms of your actual experience.

    I tried implementing an ECS to get away from the “every entity is the same, just with different function pointers” philosophy, but I ended up doing it wrong. I ended up stripping out the concept of an “entity” entirely, and only using “entity IDs” to keep track of memory ownership. A system that wants to operate on its “components” just iterates over the binary tree of pointers to all instances of that “component”. I imagine this is pretty close to what a lot of armchair programmers shitpost about ECS being for.

    Of course, if I weren’t looking at game engine structure at all, and started out with the “monolithic pile of crap” common to bespoke game engines from the NES/SNES era, I probably would have ended up with the same thing, just not organized well in terms of how the main loop gets special-purpose code to work on its special-purpose data.

    Reply
  5. Konstantin Mikheev

    I really tried ecs for several times. I absolutely have no idea why everybody are so excited about it.  It just makes things more complicated. It is better than putting everything into a god object, but is too far away from something I world consider to be decent.

     

    Thanks.

    Reply
  6. Novid

    To Mr Gaul.

    Thank you for this enlightening post. Not a programmer, but I am always fascinated how these work. The nuts and bolts of it. John Carmack re-tweeted the link and here I am, seeing how a in-house engine works best.

    This leads into the other post made by some guy at BlizzardActivisionINC. trying to come here and defend his teams process in making one of the most weak and piddling games in a generation, only hours after his company is about to cash out in the tune of northwards of a billion dollars in e-sports exploitation.

    Madam Cotto, your team and your artists had really no confidence in Overwatch at all. The only reason it even had a chance was because of its e-sports set up and your company admitted as such. The fact one used a ECS for this game, proved it. Doom built its own engine (for the reboot of the franchise and Quake e-sports) with flying colors. Even was able to use map access from the game in the past to put in the new one (not an easy feat), and while their multiplayer has issues for Doom, certain aspects of it worked better than anything Overwatch did, has done or will ever do.

    Even the Japanese Developers, which are starting to get a handle of their own engines and the ECS setups are light years of head of what Overwatch is doing, so using that ECS is fine trick cannot fly with me, when I seen what Kojima has done already with modified In house engine, Harada, Taro and others with their hard coding aspects of the ECS they use, but still have some elements from a in house engine and so on. Even Sony, has moved away from Middleware and is pushing limits that Cerny already stated they could do ON base PS4 systems!

    So please, with all do respect – dont come here touting the greatness of a front loaded, front run, over promoted, over rated, funko based, *steven unversed* art based “how ECS made Overwatch great” bs when I see much better works from those that can manipulate Middleware better than you company has done.

    A joke, and a sham, and a damn shame.

    Reply
  7. t-machine.org

    Someone asked me to comment on the ECS complaints here.

    First … Congrats on writing something contentious enough that people are talking about it. In my experience, that’s often a great way to get people actually talking about new ideas rather than simply agreeing, or disagreeing, and moving on in silence. It’s tricky to get right – too much, and people dismiss you as a childish fool. Or get distracted from the concepts you were trying to convey. Too little … and you come across as offensive, without increasing the level of discussion.

    As far as I can see, it wasn’t an article about ECS, you just needed an acronym to rant about, and make a point about architecutre in general. Strimming out the invective, I found nothing ECS-related to comment on. It reads as: “I’ve ignored the most widely-referenced sources, and instead picked some uncited, random, short, commentary by people I have no respect for. What they say is naive (because I picked people with little or no experience) .. or I don’t understand .. so I’m dismissing it. Please go and focus on writing games, instead of focussing on middleware / code architecture”.

    (even Wikipedia would have given you clearer, more concise info on how/why/where ECS is valuable)
    Broadly I’d agree with that – if you pick random opinions on the web, you’ll usually get crap. And whatever you do … most people should be making games, not engines.

    PS: I find it ironic that your whole post is about promoting a niche code-architecture approach … and then you slam the use of niche code-architectures. I think you could have made the point better in a different way – e.g. by pointing to YAGNI. Your failure to cite something as simple and fitting for your argument as YAGNI, and to instead go of on a rant about unrelated things that work, for exceptionally strong, well-documented reasons – and which over-rule your small wins – makes me question how much you understand the context you’re working within.

    Your approach is fun. I’ve done it myself, many years ago, for multiple projects. There are languages that make it cleaner/easier to do than C++. But if people don’t know the context it works within, I’d advise steering clear of it – e.g. your cry of “designers must learn C” (do you mean C++, since you’re throwing this through a C++ compiler? What bugs will you hit which seem OK in C but not in C++?) should be a burning red flag to most projects.

    Reply
    1. Randy Gaul Post author

      For some reason many readers that get offended by my posts immediately jump to conclusions about me personally. I don’t actually matter, so reasons of why I wrote the post, or whatever tactics one thinks I used to spur discussion are irrelevant in terms of the ideas presented. If an idea offends a reader, then why not attack the idea? I find it silly to make it about me, and a waste of time. Here is what it looks like to write something controversial. No glory and no thanks.

      Those opinions are not random nor uncommon. I’ve seen them a million times before, and in my opinion accurately represent common understanding of ECS. Taking time and energy to pull them out of a live Reddit thread and respond point by point is not a lack of respect, but a sign of respect. I don’t get anything good out of responding to these points. There is no benefit to me other than satisfaction at the mere concept that some readers may appreciate the argument responses. I get personal attacks for taking the time to respond, instead of any kind of thanks.

      If someone specifically want sources to be cited, then they can go look up sources for themself. Personally I don’t care about attacking sources, and it’s ironic that every time someone insinuates or otherwise says I don’t know what an ECS is they link to a source that contains links back to my blog as reference. My job here isn’t to think for the reader. Everyone can google their own sources to their heart’s content and make up there own mind. I’m here to present some ideas, no more. Maybe it would be more helpful for blog commenters to cite sources and present ideas about ECS, rather than make wild assumptions about the author personally. For example, this is what Erin did and it spurred on many thoughtful responses and made the entire webpage a better place.

      tl;dr – I’ll just quote my post: “The above section is just trying to attack acronyms and naive advice… Just take the above section with a dose of skepticism.”

      Reply
  8. t-machine.org

    How do you honestly claim: “Personally I don’t care about attacking sources” when you’ve already said:

    “online voices that argue in favor of ECS are simply very excited about following a methodology … The problem is this is a fantasy.”

    “I’ll respond to some common pro-ECS arguments seen on the internet”

    “The points raised in the thread represent very common arguments on the internet in favor of ECS…The goal here is to provide curious readers a voice that opposes popular opinions”

    …all you’ve done is attack your vague, undefined sources – that you chose – which conveniently allow you to make specious, inaccurate, statements to support a (presumably?) untenable position. That’s not constructive.

    Your claim of “My job here isn’t to think for the reader” is belied by your actions, and makes the overall post appear a deliberate plan to spread disinformation. That doesn’t help people, it drags them down.

    Reply
    1. wareya

      The interpretation of ECS that they’re arguing against are, in fact, very common on hobby game dev forums. I mean very common. They’re being rude about it, but they’re absolutely not making a strawman argument.

      Reply
  9. YAML Ain't Markup Language

    But ECS isn’t an acronym. It’s a software engineering pattern. I don’t get why you get so attached to the name of the pattern as if it being an acronym automatically means it’s bad. If it were named “bananas” your arguments would make no sense.  For example, Angular isn’t an acronym, but it’s very hyped and really bad. On the other hand, “LASER” is an acronym and lasers actually don’t suck even when their name is an acronym.

    So when you can write a post that doesn’t actually care about the name of the things that you’re talking about, maybe you’ll have better arguments. You can talk anything you want about entity-component-systems. But say that it’s bad just because its name is an acronym? That’s just no argument at all.

    (And no, I’m not offended. I’m just trying to help you create more compelling arguments so that your next articles can be better, especially because it seems most people actually got mad because you focused on the acronym argument instead of providing actual solid facts and opinions about the subject itself.)

     

    Reply
    1. Randy Gaul Post author

      Well said!

      As for software engineering patterns, I believe in the post I also said I don’t like these either. To be fair the post did contain links and excerpts to quite a lot of information about ECS, so the argument was not merely “it’s an acronym so ignore it”. It’s bad because it’s a cookie cutter methodology, at least 95% of the time on the internet for any discussion or article, not just because it’s an acronym.

      Reply
  10. Irlan Robson

    You summarized pretty well Randy.

    In some solutions on the Internet there is a lot of unecessary granularity of components. This creates a mess if not constrained correctly. It increases the probability of dependencies and defining redundant functionalities in systems. However, still, something very sucessfull might be about to be shipped.
    I think a good idea that is also less error prone is to think of an entity holding only a few fundamental and well defined data (here components) and the rest is scripted. This is much more where gameplay related stuff are going. It’s up to your team to decide whether you choose to write scripts in C or Lua or whatnot. Obviously it doesn’t really matter in the low-level side. OOP is meaningless at this point, which is a good statement seen by big projects like commercial engines.

    Reply
  11. Pingback: Exploring directions to fix game programming – NamekDev

  12. Pingback: Writing a Game Engine in 2017 | <tech.gnome.london>

  13. Ryan McGill

    I personally think it would be wise if all human beings would be aware of the flaws in our communication system(s), which is what you were hinting at:
    The label != The intention.
    Blueprints, like code, are so great at being unambiguous. If the following could represent what you might recommend:

    PeopleOnInternet;
    ECS = ECS_OriginalIntention;
    // Override ECS
    ECS = PeopleOnInternet.ECS_NewDefinition;
    // Therefore, please..
    Compare(ECS, ECS_OriginalIntention);
    // And maybe even…
    ECS.ResetDefinition(Default); // Default == ECS_OriginalIntention

    Alas, it’s not really what you were stating, but the above “code” points out the flaw that you’re warning about. People “are” using the wrong definition, all the time, for nearly every single word in existence. They are then making things happen in blissful ignorance of each-other’s held definitions.
    Then, when someone such as yourself points out the “meta” issue in language, it results in argument upon argument without getting at the root cause of it all.
    Either way, the arguments from both sides were quite educational, regardless of if I can tell what is correct. Then again, what “is” correct, whose version of “correct”?

    Thankfully, when we get down to language/programming features/syntax and closer to “code” in general, we get closer to a true meaning of “correct” – maybe. Even discussing the meta of the situation outside of a code-like language itself is just asking to be misinterpreted.

    I just wanted to point all this out. The internet “is” misinterpreting definitions, because everyone is, always. The original designer of the concept of ECS is wrong, even though he’s right. The internet is wrong, even though it’s right.
    Thus we have: Customization vs Standardization.
    We need at least one spoken language that has un-changeable standardization, devoid of ambiguity. It would make a lot of things clearer. English is grand at customization. For standardization, programming languages are the closest yet.

    P.S. I bet you could find-and-replace each mentioning of “ECS” in here, and the point would remain.

    Reply
    1. Randy Gaul Post author

      I would be quite happy with:
      PeopleOnInternet;
      ECS = ECS_OriginalIntention;
      // Override ECS
      ECS = PeopleOnInternet.ECS_NewDefinition;
      // Therefore, please..
      delete ECS;

      And be done with acronyms and glory methods :)

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *