By Cadir Lee on Wednesday, June 15th, 2011 11:25 AM PST

We’ve been heads down here at Zynga over the last few years but we’ve always been big fans of open source and open information.  We’ve learned and leveraged a lot from others and we hope that this site will be found useful in turn.  To those that have made their way here,  a hearty hello– we welcome your comments, ideas and feedback.  Zynga is a social game company and our mission is to connect the world through games — this has become quite broadly known.  Perhaps less known is that Zynga is hugely focused on technology.   The largest group of people at Zynga are software engineers and if you combine all the rest of the folks involved in software development, you’d get a hefty majority of the company.  We’ve had to develop some interesting solutions along our growth path and we’ll be sharing some ideas and thoughts along the way.  Here are some of the top areas we focus on — expect to hear more from us on these.

Games

It is probably no surprise that we focus on games but perhaps less obvious is how much we research this area.  We’ve developed our own game engine in Flash and we continually look at ways to experiment and eke out more performance, easier development or new game mechanics.  We also try to keep pace with many other game engines in the space and we’re constantly looking at new innovations.  While most of our games are simple to play, we spend considerable effort to make them fast to load, quick to respond and able to display rich artwork quickly within the confines of the browser.  We also do heavy work on mobile game engines and we are big fans of Cocos2D — so much so that we now have two of the heavy contributors to that project working here at Zynga.

Games have traditionally been an area of study for computer science because of the challenges of multiple users interacting simultaneously.  Many of the traditional solutions such as having separate servers to partition players don’t work for us.  Social gamers can play across a broad graph of connected friends and trying to chop those graphs into particular servers doesn’t work.   When you combine that problem with a massive workload, interesting technology comes out.  Our workload differs from many of the largest websites on the planet — many of them are read oriented (search, shopping, news) or write oriented (photo or video upload) but our workload is almost entirely update oriented.  Our players expect to interact with the game world and make changes as fast as they can click.  This difference challenges many traditional databases, web servers and architectures and our solutions to these problems are unconventional.

Virtual Goods

Our business is based on virtual goods so we naturally focus considerable effort there ranging from management of massive economies of hundreds of millions of players to combating fraud.  We’ve developed sophisticated models to predict demand and to understand how our players value virtual goods.  We work tirelessly to create a safe and easy purchase experience with world class customer service to make sure our players feel comfortable transacting with us.  When you add in trading and gifting, we often are managing economies that rival the scale of real world economies.  We believe strongly that technology can help our players adopt virtual goods as a regular part of their entertainment.

We think there is considerable new ground in the interaction between virtual goods and real goods.  Loyalty programs and crossover between real and virtual are fascinating business areas that we want to facilitate with technology.

Open Web

As we look to the future, we’ve been doing a lot of R&D work on using the latest browsers to make games entirely from HTML, CSS and Javascript.  We recently joined the W3C and hope to help the industry overall to enable rich social games in the browser across multiple devices.  We’ll be releasing as open source some of the tools and libraries we’ve been building including a Javascript benchmark suite and libraries to facilitate using sound and sprites across different devices.  We’re looking for others to collaborate with so reach out if you are working in this area.

Analytics

Zynga has been written about as being a metrics driven culture and whenever someone joins Zynga, they often comment on how much the company lives and breathes metrics.  As such our systems get heavy use and we’ve developed a system from very early on that relies upon deliberate structured instrumentation, massive data aggregation across multiple data centers and fast loading into one of the worlds largest columnar databases.  All this means that we can get answers about events happening throughout our games across datacenters around the world within seconds or a few short minutes.   Lest we come across as only driven by numbers, we also have significant investment in user surveys, focus groups and community forums.  We’ve developed survey and experimentation systems that allow us to get feedback from our players based on a wide set of criteria.

Cloud Computing

Because our business can take off with a single feature or a single game, scaling our infrastructure presents out of the ordinary challenges.  We are one of the world’s largest users of public and private cloud computing and we rely heavily on the elasticity of the cloud environment to handle sudden changes in demand.  We’ve developed our architecture to work more effectively in a cloud environment and to be tolerant of a heavy dose of disruption.  This robustness has enabled us to keep running even during significant cloud outages that more greatly impact other cloud businesses.  Our toolchain for provisioning and managing this dynamic environment includes a mixture of open source, commercial tools and a good does of our own development.  We continue to invest in this area but we’ve been able to leverage cloud computing to handle the extreme spikes that might require 1000 servers to be deployed in a single day.   Without automation and a horizontally scalable architecture, we’d have now way to keep up with those spikes.

LAMMP

We use a linux, apache, memcached, mysql, php (LAMMP) stack throughout our games.   We are heavy users of mysql, linux and apache but we focus our development efforts in particular on php and memcached  and have contributed back to the open source communities in those areas.  CouchBase, which is in our opinion an evolutionary successor to memcached, came about as a collaboration between a project we were developing combined with work that the CouchBase team was doing and we are running thousands of instances in production in place of memcached.  We have done considerable work on PHP and PHP extensions in particular such as APC and AMF in order to improve backend efficiency.

That is brief overview and there is a lot more so poke around and let us know what you think.  If you’re curious about something, shoot us a question.  If you’re working on something that you think we should know about or just want to get more involved, let us know.  We’re a pretty friendly bunch of folks.  We play games all day.

 

28 Responses to “Welcome to Tech blog at Zynga”

  1. Karthik Kumar Viswanathan Says:

    Love it!

  2. Amar Arsikere Says:

    Excellent article! Provides a succinct and foundational background about Zynga Engineering

  3. Ami Huggett Says:

    Awesome :)

  4. Brady Stevenson Says:

    Wow. Very well written. I don’t think a lot of people consider Zynga a cutting edge company because they make games, but this is great in raising awareness of what challenges they actually face. Good stuff!

  5. Amar Arsikere Says:

    Good to see the traffic flow to the site!

  6. Vyas Mohan Thottathil Says:

    The other equally awesome side of Zynga Games !!

  7. Gautam Chakravarthy Says:

    Great forum to showcase the versatility of Zynga!

  8. Sarfy Says:

    Our team is excited to be a (tiny) part of your success! We love you!

  9. Benny Says:

    Did an engineer write this or a marketing/PR person? Doesn’t offer much tech insight, nor does it have an authentic engineering voice.

    Way to not keep it real :(

  10. Michal Todorovic Says:

    Thanks Cadir! A great overview of who we are on the engineering side.

  11. cadir Says:

    Benny, I wrote this and I’m an engineer. I appreciate the compliment though that I can write like a marketing/PR person. This is a welcome note and an introduction — stay tuned for more.

  12. Vijay Patel Says:

    Thank you Cadir. I am a confusing on one singular point..

    In the internet it is saying that you are CTO of Zynga but I also have met two other chaps who are saying they are the CTO, and another one who says he is CTO in the Quora forums.

    Will you kindly unclearing my confusion.

    regards from India – Vijay

  13. Xiang Ju Says:

    Very interesting that Zynga started in 2007, and their FIRST technical blog post comes in 2011…

    That said, can’t argue with the success $-wise

  14. Vijay Aswadhati Says:

    Very good overview of the tech side to Zynga. Congratulations!

  15. Matt Manuel Says:

    @Vijay: Each studio (Game, Platform) typically has it’s own CTO. And Zynga overall has a CTO: Cadir Lee. So there are multiple CTOs at Zynga, most at the studio level.

  16. Ketan Rathod Says:

    We are like a swan, cool and calm over water and constantly paddling under water

  17. Sergey Z Says:

    I no knowing why comrade Yuri invest in rubbash like this zynga company and other gaming company in california.

    How to have many CTO. Chief is only 1 in company.

    in most company like google these people just called tech leader not CTO.

    also PHP is old and borring.

  18. Jackie Says:

    LAMP Stack. Innovative and brilliant.

    In other news, Zynga decides to use Scotch tape for taping stuff, MP3 standard for listening to audio, and Kleenex brand for tissues.

  19. Daniel McCaffrey Says:

    Great stuff!

    @Vijay and Sergey ….Cadir is the Zynga CTO , as a title and officer of the company, and very much an engineer at heart and practice. There are other CTO “roles” usually studio based.

  20. Satyajit Says:

    “we’ve been able to leverage cloud computing to handle the extreme spikes that might require 1000 servers to be deployed in a single day.”

    Just curious to know which 3rd party cloud services do you leverage for these extreme loads?

    Looking forward to hear more.

  21. Pramod Patel Says:

    The comments here sound like a love-fest from Zynga employees. At least the first 10.

  22. Shay Pierce Says:

    Very cool to see Zynga start to publicly support open-source initiatives, especially ones they’ve been benefiting from…

    I heard that Empires and Allies used as3isolib, an open-source AS3 library for rendering isometric graphics. Is that true? Was it used out-of-the-box or with modifications?

  23. Billy G Says:

    the technical insight on the followup comments on this blog are very awesome.

    great job zynga. you are a true “engineering company”

  24. Frank Says:

    I’ve met many Zynga CTO’s too. They make it a point to say CTO at Zynga, versus CTO of their studio. Hence the confusion.

    Cadir sounds like the real CTO.

    cadir ftw

  25. James Rhys Says:

    If you are, I’d love to see any challenges you’re facing in transitioning your games engine to HTML5.

  26. Khan Says:

    Use Chrome, can not even get Frontierville to load for weeks. I get the accept/respond gift page and then it just sits there. This HAS to be ZYNGA issue not Adobe. It used to work ZYNGA. I can get Castleville to work just fine, so not Adobe.

  27. William Davis Says:

    canon lens
    Equipped with an Image Stabilizer and high zoom ratio, the Canon EF 28-135mm standard zoom lens delivers sharp, natural-looking pictures.
    http://digital-camera.co/canon-ef-28-135mm-f3-5-5-6-is-usm-lens/
    canon lenses

  28. maillot real madrid manche longue Says:

    This is one awesome blog.Thanks Again. Much obliged.

Leave a Reply

Connect with Facebook