When to out-source development vs building it in-house

“Should I build an in-house team or out-source my project to a software development shop?”. This is a question I’ve been asked many times as a technology leader, by founders and managers looking to build or improve software products.

Earlier in my career I would favor keeping software development in-house for most projects, but I quickly learned of the many challenges it can bring to a project. In-house teams may bring strong engagement and dedication to your project, but will sometimes lack important skills required to get the job done.

There’s no black and white answer to the “hire or outsource” question in software development.

So what are the pros and cons of each strategy? What are the risks and challenges? What will it cost you? Below we’ll discuss how to choose the right strategy for your company.

In-house software development team

The great thing about having an in-house team is that the people you hire will be dedicated to your team and project. Their interests are aligned with your company’s thus your project will always be a priority.

Whereas an outsider will always view you as a vendor and might not fully offer you a product as per your need.

Pros:

  • High engagement since most interactions are face-to-face;
  • Both your interests are aligned because their success is highly correlated with the company’s;
  • The knowledge acquired through months of development stays in house and generates valuable IP;
  • You’ll have more room for product iterations and pivots;
  • No language barriers, timezone issues and cultural differences that can often slow down projects;
  • More control and an easier time managing all the moving pieces of the project.

Cons:

  • The hiring process is often slow and costly (recruiter fees, job ads, training, etc);
  • Forces you to focus on other business functions like operations, culture, HR and legal, which is time consuming and requires additional staff;
  • Can get expensive with salaries, benefits, taxes, raises, etc;
  • Tough time hiring in a competitive market where finding talented engineers is hard.

When you should build or augment an in-house team:

  • Your product is still full of assumptions and you anticipate a future iterations and pivots;
  • You have a technical lead who can help recruit the right talent and manage the team;
  • You’re okay hiring staff to ensure operations, culture, HR and legal needs are handled properly;
  • You’re a startup looking to generate IP and attract investors.

Out-sourcing to a development shop

Pros:

  • They have all the resources needed and ready to start;
  • Better time estimates and planning since they’ve done it many times before;
  • Less expensive since recruiting, training and on-boarding of employees is not necessary;
  • Faster delivery to market when all requirements are scoped out correctly;
  • Learn effective strategies, practices and tools from experts who’ve done it before.

Cons:

  • Less room for future iteration and pivots;
  • They tend to push for long-term commitment, so incentives are often not aligned;
  • Planning sessions can take months before development starts;
  • Poor communication since dev shops are sometimes located in countries with different time zones and cultures than yours.

When to out-source development:

  • You have a clearly defined scope of the product features (on paper, not in your head!);
  • You’re not expecting many product iterations;
  • You have the budget to pay for unexpected out-of-scope features and maintenance costs;
  • Your project has a tight deadline and talent is not readily available.

Conclusion

While outsourcing to a software house may seem more cost-efficient, having in-house experts hired will definitely provide more benefits in the long term. It’s often a good idea to mix both approaches at the same time depending on the project, while considering budget, deadlines and scope.

It should also be noted that if your company is looking to become the next Airbnb or Facebook, my recommendation is to invest in an internal team. Good software products go through dozens of iterations and pivots in order to stay competitive; having an engaged and committed team is important.

What do you think? What did I miss or forget? Have you or your team experienced some of the challenges this article addresses?

I’d love to know.

Why we ditched Unit tests for Integration & End-to-End tests

With the rise of agile development practices like continuous integration, development teams have struggled to find the right balance between unit, integration and end-to-end tests. Most developers settle for the 70/20/10 rule as promoted in this Google Testing article which splits testing to 70% unit tests, 20% integration tests, and 10% end-to-end tests.

For some spending 70% of their testing efforts on unit tests is not optimal; this was the case for our team at Systelos, a SaaS platform that allows financial advisors to guide client behavior at scale. We tried it for a few months and it didn’t work, so we ditched unit tests in favor of integration and end-to-end tests.

The problem

As a team, we wanted to be able to answer questions like is our application working the way we want it to, or more specifically:

  • Can users log in?
  • Can users send messages?
  • Can users receive notifications?

The problem with the 70/20/10 strategy which focuses on unit tests is that it doesn’t answer these questions and many other important high level questions.

Unit tests are too isolated to test real user stories. They are fast and isolate failures really well, but don’t provide the same confidence integration and end-to-end tests do. Especially with a frontend heavy application like ours, where the user’s behaviour is complex and unpredictable.

Furthermore, most issues reported in our JIRA were a result of miscommunication between multiple UI components and backend services. When incorrect information was presented to our users it was usually because a service didn’t access the required information from the right location, or because different frontend components didn’t work together properly.

Our solution

Since unit tests didn’t have much impact on the reliability of our application, we needed a better testing strategy to help us answer high level questions like, can a user still send a message after all these code changes?

After close inspection, we noticed the small amount of end-to-end tests we had did a better job of spotting issues we hadn’t seen before; they proved to be valuable in our quest for reliability. This prompted us to to flip the pyramid on its head and focus on both end-to-end and integration tests.

Our new strategy was to split our automated tests into 20% unit tests, 30% integration tests and 50% end-to-end tests. With the help of testing frameworks like Jasmine, Protractor, Sinon and Karma, this new strategy helped us catch 95% of issues before they made it to production.

20% unit tests

We still believe unit tests provide many advantages to our team; they run fast, provide great isolation for failures, and are much more reliable than integration or end-to-end rests.

Because we now only use 20% of our testing efforts on unit testing, we’re picky about which functions to test. We only focus on functions that have a high probability of breaking when the logic is incorrect or when it receives wrong inputs.

30% integration tests

Integration tests are tricky to implement especially on the frontend because larger functions tend to interact with many different services and APIs. We use Sinon.js to spy, stub and mock external services since they’ve already been tested. And just like our unit tests, we only focus on high reward functions.

50% end-to-end tests

We focus most of our efforts on end-to-end tests; they test the whole application from start to finish. They ensure that all the integrated pieces of our application function and work together as expected. They simulate real user scenarios, essentially testing how a real user would use the application.

We use Protractor to control the browser and simulate a client nudging and starting a conversation with their advisor regarding an update about a life change. Running multiple tests like these to test every single functionality within our application helps us ship code confidently.

There are however hurdles we had to overcome in order to make this possible within our continuous integration environment where code is released every week. Some of them include:

  • End-to-end tests take a long time to run. This can be a problem when you’re trying to deliver code fast (continuous delivery FTW!).
  • They sometimes crash for no reason. The browser may treat long running requests as failures, or another developer may have changed the IDs in your HTML without updating the tests.
  • They take a lot more time to write. Since user actions require multiple steps, you need to be extra careful managing your IDs, adding timeouts, simulating external devices and browser capabilities, the complexities are endless!

We took several steps to solve these issues, and I’ll be sharing them with you in my next blog post so stay tuned!

Conclusion

Finding the right balance to how much unit, integration and end-to-end test to write is not easy. Every team and application is different and requires a different ratio. A good way to identify the best strategy is to analyze and reflect on questions like which tests within your codebase have made a big impact? which ones saved you from having your application crash on production? For us, it was clear that end-to-end tests made the most impact and found the most bugs, with integration tests coming in second.