A view on the essence of Data Center colocation and hosting business

I met this gentleman, lets call him Y, a couple of years ago and we as a group of twenty people had just completed a tour of a data center which is part of the programme of a data center conference. We are having a cocktail reception sponsored by the data center service provider that we had just visited. This data center company that hosted us is about two years in operations.

We are talking in groups of threes and fours. Y suddenly said in our small group this host’s data center is not good, his well known and decade old company’s data center has unbelievable this much of documentation and they are updated so regularly. I am no doubt that his company people works very hard and put in lots of effort, but what does it have to do with three-month-into-the-new-job him. But I digress.

 

Good start

The host wanted our honest to good opinion for them to improve their company’s data center operations and grow into a solid member of the data center industry we are all in. Putting more paper to a documentation is an incremental process and we should focus on what is important to a young data center company or even to an old data center company.

We can always learn something new when we visit a data center. In fact, the data center industry is itself very young. I can only praise this data center service provider that is willing to host visit from the same industry so that we can all share and learn together.

 

We are only proud of our past, and it is because no fault came to pass

We can have the most documentation in the data center industry, but outage will happen and it can be human error, battery problem, equipment degrading to below the designed load/capacity, hacking from  outside the data center, and forces of nature beyond the data center control.

Most people (clients and data center service providers alike) will have the following list:

  • Location
  • Expansion i.e. scalability (of power and/or space)
  • Availability of power
  • History of reliability
  • Resiliency of the data center infrastructure design
  • Telco carrier availability
  • Financial stability
  • Fault Response and Recovery

and the list goes on and on.

No data center, after a long enough time, is able to stay free of fault. It should plan and practice how it will react and recover from the fault.

Those factors are important, but they are attributes that comes from meeting the customer needs and reducing the amount of worrying by the customer.

 

 

We should always focus on the client

What is the essence of Data Center colocation and hosting business?

Let’s go back one step. A data center colocation service provider’s main business is to serve client that put its mission critical IT gear in the colo data hall.

From a enterprise client or the vantage point of any client of a data center, what makes the client key personnel like CIO less worrisome about their mission critical IT system sitting in a third party’s space?

 

From the words colocation and hosting, it means we are host and our client are our paying guest. We have to earn the trust of the client, and client staying with us (like good comment on tripadvisor or booking.com does for a hotel) whether its incident free period or during incident event itself whether we come clean and react in a professional and effective manner bringing all our resources (including our support vendors) to rectify the fault (like hotel replying professionally on tripadvisor or a hotel booking website).

 

 

Where all big guys came from, stay young

Small young data center is where large data center company comes from. We can be big, but we are still attentive to details and respect someone trying very hard to do a good job.

Let us not forget we are once small and we are all in a business to be a good host. Let us share and learn from each other, and provide better service to our clients.

And it is good to think young. The client is demanding and IT keeps evolving, or we should approach and ask the client, for more knowledge about their IT side of things. More on this area in future post.

 

Share, learn, altogether

We as an industry is young. I am surprised to find that the data center players in most countries are not organized into an open group to share and learn. Engineers, doctors, hotels (4/5 stars, boutique), all have their associations and they do share updated development of technology and new service capability.

I for one will like to see a open data center group for all the data center service providers within Singapore get together and learn and grow together. Perhaps Singapore IT Federation should have a data center chapter? Or how about South East Asia? or Asia? I mean even the telecommunications carriers have Asia Pacific Carriers’ Coalition.

 

Reference:

This is a good article about hosting sales. http://www.datacenterknowledge.com/archives/2010/02/24/selling-hosting-in-a-sophisticated-market/

http://www.datacenterknowledge.com/archives/2015/12/09/whats-driving-change-in-the-colocation-data-center-industry/

 

 

Advertisements
A view on the essence of Data Center colocation and hosting business

Sustainable Data Center involves both clients and Data Center service provider working together

green-server

While previously working on the Singapore Green Data Centre technical sub-committee and also in the line of data center business in Singapore, Australia and China, I discovered something as simple as electricity charges has a different reaction from the different clients even within the same country. This is due to differences in treating the electricity usage with the rack/private suite space that the client leases.

The IT user management has for the most part been responsible for management and cost of outsourced data centers. They are now looking at the operations expenses and it often comes up that what have they been paying as part of leasing the private suite or those racks.

In China, as I mentioned in my post number 2 on my view of the China Data Center market (2016), there is a two tier data center market. The hosting and shared co-location customers rent by the rack and wanted the electricity charges bundled at a flat rate and wanted it as a total cost and not separately shown in the invoice.

For clients who lease private suite from wholesale data center service provider, some will opt for flat electricity charges bundled in the invoice because they in turn sub-lease the data center suite space on a per rack or even per 2U server manner and wanted it simple.

The data center service provider is made to do what is simple, which is to take say 1RMB (for example, the average of peak+low of some cities are different) x 20Ampere (the contracted capacity) multiply by 24 hours multiply by 30/31 days to arrive at the electricity charge to bundle into the per rack or lease space charge (say private suite can house 50 racks, then its 50 x 20 x 24 x 31).

In most time, the data center service provider have a bit of a margin earned from all the customers who don’t use up to capacity, and there is no motivation to improve the power efficiency as the actual power consumption is not linked to the power charges bundled into the invoice and it does not apparently benefit the data center service provider.

This bundled-all-in-pricing will cause the client to miss the whole picture and longer term sustainability of the data center and the data center business itself. But this should be changing, sooner than later.

There are data center clients and data center service providers that practice a more direct pass through model.

This is how it works:

  1. The data center suite is still build to client requirement and handed over to the client.
  2. Private electricity meter at the sub-board for the client.
  3. The agreement struck is such that the utility supply company’s electricity bill is handed over as part of the invoice, and the client pays the portion that is metered at the private electricity meter.

Some variation to number 3 is that there is a agreed PUE factor to multiply to the electricity charges that the client pays.

Paying actual usage sounds logical. There is a problem which is under-utilization of capacity that the data center service provider has designed according to client specifications and yet the client did not fully utilize. The power grid and/or the power utility company will impose charges (penalty) for unused capacity.

This has to be viewed in two parts:

  1. Firstly, the infrastructure component whether capacity is fully utilized or not should be fully factored into the lease space rent
  2. Sub-optimal utilization of the contracted utility capacity, this needs careful prediction of power capacity on a per month/quarter/year depending on the power utility company pre-requisite on adjusting the requested for capacity, but this can be built into the client agreement

 

Ultimately, the data center service provider should not view or consider to earn from utility charges, firstly most countries forbid it as part of power supply regulations and secondly, the client will view it negatively and it will affect future business.

Understanding and achieving both a transparent and more direct linkage to the utility supply will enable:

  1. Energy efficiency planning and projects by the data center service providers
  2. Energy efficiency planning and projects by the data center service providers
  3. The best of them all is Energy efficiency planning and projects by the data center service provider and the client.

From managing the energy efficiency all the way from the utility supply into the building, through the data center service provider data center infrastructure, and through the sub-boards onto the rack PDUs into the servers, a lot more can be achieved for energy efficiency that will reach that most important driver of all: cost savings and sustainable business for both the client and the data center service provider.

Sustainable Data Center involves both clients and Data Center service provider working together

A view on the China data center market – part 2

qiandaohu-data-center-u=971122560,122887307&fm=21&gp=0

The picture above is a data center in the scenic HangZhou QianDaoHu district which Watone (www.farvin.com.cn) operates for Aliyun. It uses the lake water to cool the data center (design PUE of 1.3 under full load).

This will be the second post on a series of posts by me and a colleague on the China Data Center market. You can find the earlier post to this series on linkedin under my posts or via newwitblog@wordpress.

Carrying on from post number 1. Please bear with me if there is repeated content from post number 1 that are here.

  1. Government

Besides IDC license issue, another issue that worries foreign firms especially data center is Internet control and censorship, such as through the great firewall of China (“GFW”). However, it affects content to be accessed from outside of China and unless it affects local colocation data center business, no local data center service provider are known to be very concerned with this issue. My advice is to play it cool, if you want to join the local competition then don’t worry too much about it.

While there is no specific initiative to attract foreign data center service provider into China, there is no deliberate attempt to discourage such foreign data center operator. Chinese government, like other Asia Pacific economies, is not well aware of the data center industry. Chinese government officials are very well versed with smart city, cloud technology, big data analytics but are probably only aware of industrial property, industrial parks, so give them a bit of leeway. The local government officials, at county level, will be open to discuss about data center development as it brings investments and also will employ highly skills workforce which is especially important to tier-2 and tier-3 city/county government official because both investment and employment of highly skills workforce are part of their KPIs. Talking to them will also let you have an idea whether there is enough power and water for the places you are interested in. Another way is to phrase your key words as “cloud data center” (“云数据中心”) as you can say that your data center will be ideal to house cloud technology, cloud data center they understand amazingly well.

There are lots of positives in terms of support from the Chinese government. They encourage investment in data centers, cloud, and big data analytics. The government has issued directives that government system and data can be outsourced to third party to manage (http://www.gov.cn/zhengce/content/2015-01/30/content_9440.htm). This will be a good guarantee of base load for any government built data center that you may be interested to look at.

From 2010-2014, new data centers in Beijing is estimated to have grown at 29% per year. That is a very fast pace and in the next five years it is still predicted that total data center space for whole of China will continue to grow at double digits.

The Chinese government’s Twelve Five Year development plan (2011-2015), green data center, cloud technology, big data are encouraged.  From 2010-2014, new data centers in Beijing is estimated to have grown at 29% per year. That is a very fast pace and in the next five years it is still predicted that total data center space for whole of China will continue to grow at double digits.

In the to be finalized thirteenth Five Year development plan is expected to push for intelligent cities, cloud technology, Internet of Things, Big Data Analytics amongst the high technology development areas of focus.

Government support is there in another form. A lot of the data centers are built and are empty. The government has the funds (through local government investment companies facilitated by government financing) and has built data centers. For example, the NingXia province ZhongWei city government is building three data centers for Amazon Web Services. The ChongQing city government has funded a substantial chunk of the data centers cost for two data centers for Pacnet. There are still several hundreds of those data centers that are owned and managed by government BUT are not solely for government use themselves.

The central Chinese government has realized that there are too many data centers built by the government, it has issued directive that it will centrally plan and control such large data center build from 2015 onwards.  The central Chinese government has also start to encourage new data center build to be green, with PUE of 1.5 or even lower. Beijing city in 2014 it had announced any new data center must have PUE 1.5 or below.

 

  1. The local partner/s

Whether you take the “IDC” license route or not, you will need a local partner as joint venture partner or as sales channel that provides the ISP connectivity.  A well-established company is much better than an eager new company. The Chinese business and government communities recognize and prefers stability. And there is a requirement in the IDC license application form that requires the local partner to have at least a few years in operations.

Choose a well-established company that has presence in more than one of the three network hubs (Beijing, Shanghai, and Guangzhou) in either Beijing or Shanghai first, preferably with the city that you will establish your first data center as you will reduce the work to get it off the ground. Don’t register in Beijing and then go to Shanghai for your first data center, you will waste time and effort knowing two sets of people in two cities for the first data center. Conquer one city at a time.

There is this reality, be prepared you will need to work with different local partner in the different cities you want to build your portfolio of data centers. This is because there is only a handful of data center companies that owned data centers across the multiple cities in China.

I have met with at least 8 different ISPs/ICPs/managed IT companies and data center companies in Shanghai, with varying willingness to partner for either their existing data center or build new data center. You will need to check out their stories, their finances and this takes time and effort. Some large state owned enterprises have an IT subsidiary in Shanghai, Beijing and nearby provinces that may be worth talking to. However, their bureaucracy, internal politics and slow process will test your patience. However, successful cases of Fujitsu and Pacnet in the reference cases section are examples whereby they have established joint ventures with large state owned enterprises’ IT subsidiary to jointly own and develop data centers. These joint ventures get the help and experiences of large state owned enterprises to deal with the administrative and tedious process of IDC license and dealing with their fellow public administrators for land, power, water, various licenses (e.g. fire safety, building safety, construction safety etc) much more efficiently.

 

  1. Having Carriers and good network connectivity in your data center facility is very important

One important part of a data center, for China and possibly other developing economies in Asia, is the Internet connectivity. It is most common for a data center client to evaluate if the data center facility has any of the carrier connectivity. The clients are used to having this carrier connectivity as a full package. Unlike other developed economies or those with liberalized telecommunications market where carrier neutral data center facility allows customers to choose the telecommunication carrier to use, it is the other way around in China. The clients look for data center facility with the telecommunication carriers in mind.

It is also why it is important for local partner especially one that has the ISP/IDC license which this partner will bring the connectivity into the data center facility.

And the quality of the network connectivity, and the cost your partner can get them (list price from telco means no competitive advantage), and the quality of the network route (if your Shanghai data center customer’s server response takes the path to the Beijing network hub and back, then your customer will complain and may move their business to a facility with optimized network routing).

 

  1. GB 50174 Chinese Data Center Standard

Unlike the Telecommunications Industry Association (TIA) ANSI/TIA-942-A Telecommunications Infrastructure Standard for Data Centers which is an American National Standard (ANS) that is mainly about telecommunications cabling system standard with appendices on data center, the GB 50174 is written specifically as a Chinese National Data Center standard, it references other Chinese national codes such as the building and fire safety standard.

The GB 50174 classifies data centers into class A, B, and C. And these roughly corresponds with Tier (Tier is now called by TIA to be Rated) 4/3 and T2, and T1. Why the class A matches tier 4 and 3? It was a deliberate decision as T3 is not enough and T4 is too much. There is no mention anywhere in GB 50174 for 2N backup generators unlike the Uptime Institute’s description.

This can be confusing for the foreign data center service providers. My advice is to build to class A of GB 50174, and you will not invest as much as a Tier 4 which is what you may think you are doing. Class A sits between T4 and T3.

You will find that majority of the data centers of good repute and quality build are of class A are 2N UPS, with N+1 generator backup, N+1 chiller support. This is the norm. Any variations to have 2N on generators and chillers add to your cost and reduce your cost competitiveness. You may want to be different for good reasons, but try to understand what the customers are looking for.

Go and visit the local data centers, observe and don’t state or make comments about superiority of your overseas data center. As a side note, the Chinese banks do have a 2+1 (2 production data centers + backup data center) “两地三中心” data centers operations strategy as mandated by the central bank, so they rely on multiple data centers for availability of their system and data and not on any particular data center resiliency design and feature. Second is the Chinese businessman are proud of what they have built. It is always better to make positive comments.

Second thing about Chinese data centers are that the technical and operations folks hardly move from one company to another, and data center design and advisory companies are few and the technical people knows their data center but they don’t know about others and the data center designers do not carry over their experiences from one project to the next (more on this in later post in this series).

 

  1. Huge Local Demand and Low Foreign Demand

There appears to be a two tier market in the key data center hubs.

Generally, large enterprise customers and BAT require class A data centers, and this is the first tier customers. The retail colocation data center usually attracts and host on price which meant these data centers are sometimes class A or class B claiming to be class A.

BAT drives a lot of the demand and BAT have dictated the kind of data center build they want. For example, Tencent is very specific and theirs micro-modular data center looks almost identical be it in Tianjin, Shanghai, or GuangZhou. Same goes for Alibaba and Baidu. And they take up a lot of the data center space in the key data center hubs.

It is the huge demand of data center space from within China that fuel the growth of the China data center market.

Foreign companies also demand class A type data centers. Most foreign companies’ look for data center space around Shanghai, where it is also the headquarter location of choice for most foreign companies that are in China. The most common complaint of foreign companies looking for or are already co-located within a data center facility in China is the lack of appreciation of the SLA and incident response time and incident escalation process, coupled with a language communication issue, there are a few data centers that caters better to foreign companies looking for a co-location data center space especially in Shanghai.

There seems to be no huge requirements for data center space in China from the foreign companies with operations in China. The well-known exception is Amazon Web Services, which uses three data centers in Beijing, and ZhongWei city government is building three data centers meant for Amazon Web Services.

 

  1. Profile of Data Center Companies

A data center building with data center infrastructure is capital intensive. Compare to the real estate business, it does not have high barrier to entry if you can secure land use. However, it ties up a lot of capital as generally the data center infrastructure adds 5-7X more capital needs than a data center building shell.

I classify data center service providers as follows:

  1. Telcos
  2. Data center service provider
  3. Managed IT / ISP / ICP / Cloud companies
  4. Equipment vendor
  5. Real estate developer
  6. Investment arm of government

The first group has three dominant players and there are a few other broadband service providers with data center spaces. For the second group, it is a very small list, with 21vianet, Centrin, GDS, SDS amongst them.

For the third group, they usually don’t have huge capital to expand beyond their existing one or two data center with a few exceptions (e.g. Sinnet, DrPeng, Watone), and are seldom having data center that are standalone dedicated data center facility.

For the fourth group, Inspur (浪潮), which is well known for IT servers, is a representative company.

For the fifth group, the data center is more likely to be a dedicated standard alone building, situated in the mix use office/IT/high tech park or compound that they have built and owned (e.g. RangeIDC at Hebei LangFang city’s  润泽国际信息港). While building a data center is capital intensive, the low barrier to entry meant many local data center players in many cities in China. There are 26+ IDC licenses given out in Shanghai alone.

The sixth group, ultimately looks for a company from first to fifth group to take over and operate for the mid to long term. For example Tianjin Eco-city’s data center building, the ChongQing cloud development zone’s data centers.

Another piece in this series will talk more about the data center players in China.

There are a few loose groups of data center industry organizations:

  • The China Data Center Committee group (http://www.chinadcc.org/)
  • The China Data Center Alliance (http://www.dca.org.cn/). This is organized under the auspices of the Ministry of Industry and Information Technology.
  • The China IDC Circle (http://www.idcun.com/)
  • China Green Data Center Advancing Federation (http://www.jifang360.com/union/)

All of these groups organize their activities in Beijing. Some of them do chat regularly via mobile chat tools such as Wechat.

 

  1. China is not a single homogeneous Data Center Market

Beijing, Shanghai, and GuangZhou. These are the key Internet hubs of China, and are the right places to begin the first data centers. These will be described further in the furture posts in this series. These are also the Internet exchange hubs both Internal within China, and also the Internet gateway of China with the rest of the world, mainly to Tokyo, Hong Kong and rest of the world.

There are also important new developments in China on secondary data center markets, some are in support of the above three mentioned hubs, such as the proximity of cities of Tianjin and LangFang to Beijing, KunShan to Shanghai, and ShenZhen to GuangZhou. There is also new and promising places especially western part of China. These new and promising areas include ChongQing/ChengDu, NingXia ZhongWei, and Suzhou, HangZhou. More on these in future posts in this series.

 

  1. Pitfalls to avoid

The following is a non-comprehensive list of pitfalls to avoid wasting effort and time:

  • Avoid jumping in to get the IDC license. Go find a legal firm in China that can answer fully and clearly.
  • Avoid saying you know China, even it is your tenth visit. A visitor will never know a place with a different working language and a culture as steep as China.
  • Avoid introducers or brokers. If you must use them, use them once and pays the introducers fees and deals direct after the first contact.
  • Avoid going over too many levels, as this is considered a virtual face slap to the official who is bypassed, to get the attention of someone to address your issue. The right balance is hard to strike and wrong door is a bad move as well.

 

  1. Things to do

There are good things to do when coming to assess whether-to and where-to have a data center in China:

  • Get an advisor (firm) who knows the data center market. Going it direct will be more effective only if you are a big name player and have been looking at and talking to local stakeholders for a period of time (e.g. Equinix that bought Asiatone for Asiatone portfolio of China and Hong Kong data centers, reference 6).
  • The number 1 problem that foreign data center player face in China is the language barrier. All official documents are in Chinese. While there are countless professionals and officials that are well versed in English, ultimately you will have to grasp or translate to the official Chinese language for any documentation. Get good bilingual senior managers.
  • Set up a local presence. For a data center service provider, the Chinese customers will only talk to you if you show the money and be present. Deliver is when the data center asset is fully operational.
  • Consider to pare down your data center service offering until such that you don’t need the IDC license
  • Get good partner that complements your data center service offerings, e.g. offers Internet connectivity, get customers and manages channels, etc.
  • Talk to city official, the highest rank but also the right level. There is no point talking to the city party secretary while you overlooked the city vice major in charge of high tech park. The right balance is hard to strike and wrong door is a bad move as well.

 

What’s to come in the series on the China Data Center Market

The rest of the series, not necessarily in the order shown, will look at the following topics:

  1. Current China Data Center Market players
  2. Beijing Data Center market
  3. Shanghai Data Center market
  4. GuangZhou and ShenZhen Data Center market
  5. ChongQing+ChengDu Data Center market
  6. and other China cities that is considered promising growth markets
  7. Technical Advancement in China Data Center Market

 

Reference (some of these links are in Chinese):

  1. http://news.idcquan.com/news/59946.shtml
  2. http://www.gov.cn/zhengce/content/2015-01/30/content_9440.htm
  3. http://www.chinadcc.org/
  4. http://www.dca.org.cn/
  5. http://www.idcun.com/
  6. http://www.equinix.sg/company/news-and-events/press-releases/equinix-completes-acquisition-of-asia-tone-expands-data-center-footprint-in-asia-pacific/
  7. http://www.chinatraining.com.cn/cms/media/news/2015/3492/0906091320111.pdf
  8. http://www.wtoutiao.com/a/1093756.html
  9. http://www.gov.cn/zhengce/content/2015-01/30/content_9440.htm
  10. http://www.chinatraining.com.cn/cms/media/news/2015/3492/0906091320111.pdf
  11. http://www.cfc365.com/technology/data%20center/2013-10-11/11306_3.shtml
  12. https://www.aliyun.com/zixun/content/2_6_92536.html
  13. http://www.cfc365.com/technology/data%20center/2013-09-04/11004.shtml

 

A view on the China data center market – part 2

The greatest steal of them all, and why we should do it too

A man came to a high tech conference and exhibition show and said to the security manager, I am going to steal the best things from this place. The manager was appalled but he couldn’t act on his feelings given that the guy has not done it yet. So he puts a detail that follow the guy. The man was at the conference and exhibition the entire day, talking to people, taking the freely offered brochures, getting name cards that people handed to him. When the security manager stopped this guy at the exit of the conference and exhibition hall, he asked him directly that he managed to stop him from stealing his prize, and the man said, “I have accomplished more than I expected!”, he followed on to say, “Don’t worry sir, I am here for the knowledge and industry updates, so your job is safe.”

This story has been told multiple times by others, I am just rehashing it to tell you the same.

In our data center industry, we do have conferences and granted that many of the technologies involved do not change on a day by day basis, but there is always something interesting going on in someone’s data center or somebody’s laboratory. I especially love conferences that brings the IT directors, CIOs, data center operators, product vendors, consultants, the more the mix the better the interactions and things we can learn from each other.

Furthermore, some data center practitioner may share a problem that have plagued your data center site or give you new ideas to try out. Last but not least, we hardly connect beyond our group of suppliers and definitely seldom with other data center operations people. Get out there and network, you need to face more than equipment and display screens. If you don’t try, then you lose the opportunity to something new.

Go ahead, register for a conference, get out of your comfort zone, and exchange ideas.

The greatest steal of them all, and why we should do it too

Sharing on Data center site selection or evaluation

 

DataCenterPg12

First off, I have to say neither data center site selection or site evaluation is a specialty of mine or a frequent happenings for me.

I share a few stories with regards to site that I looked at, most of which happened to me when I was in China.

Data Center site selection, and Data Center site evaluation are two distinct processes. The former deals with selection a green field or brown field (usually by a first party, i.e. data center developer/owner type), while the latter deals with evaluating a facility is suitable for use as a colocation data center (usually by a client).

This is a sharing via stories and not on concrete steps or checklist to follow when doing site selection or evaluation. The criteria that I look at are based on the TIA 942 environmental criteria.

 

No such things as a Constant Site Profile

A site will undergo change, albeit most time slowly but sometimes it can be very fast. Like the old data center at west Sydney that was shared, something came along and changes the surroundings which is beyond your control.

A data center site that is suitable when you last reviewed it, however it will change after a day, week, month, year, decade.

tianjin-incident-20150814024919762.png

In China Tianjin city. The Tianjin Binhai New District explosion that happened on 12th August 2015 at 22:50pm local time, the explosion took the lives of 165 souls. It also affected data center facilities around it, some are about 1 to 2km away. Those data centers within this 2km circle includes Tencent, Standard Chartered bank, National Supercomputer centre, China Hewlett Packard. Some data centers that are further away but still within Tianjin are Sohu, 58TongCheng, 21Vianet, China Telecom and China Unicom.

The explosion took place in a dangerous goods warehouse that was built in 2013, which was well before the data centers that are already in Tianjin Binhai New District.

In Singapore, a data center I was well acquainted with 16 years ago is in a small industrial complex with six industrial buildings. It is right next to a big housing estate and about half a mile from the subway station.

Three of the buildings are well suited to be data centers as they have good floor loading factor and pretty good floor-to-ceiling height. The industrial complex suffers 16 years ago when two major tenants moved out of those three buildings that they leased. It was good for the industrial park owner and a couple of data center companies that these spaces became available.

However, this entire industrial complex changed hands three years ago to a REIT, and the new owner plans to covert three building closest to the main road to draw in public crowds with food courts, pedestrians friendly shops,  retail and food and beverage tenants. A big no-no to a data center for high human traffic to come close by. A reassessment of options should already have been done.

And, a criteria important to a MNC, maybe totally unacceptable to another. And it takes time for one to consider the particular country or environment one is in. Two stories came to mine. First is once I was helping to provide consultancy to Gauteng provincial government data center, and they said what sort of criteria should they looked at to enhance their data center. When I read online about the notice they put up at the gate of the data center building, I was surprised to read “no legal or illegal weapon to be brought into the data center building”. Singapore is pretty safe and illegal firearm is nearly impossible to obtain and penalty for carry firearm is minimum life imprisonment here in Singapore. Well, they meet TIA 942 Tier 3 (now call Rated 3) describes a bullet resistant walls, windows, and doors especially for the security/pass exchange room.

Another story on the criteria important to some and unacceptable to another is the being far from an embassy criteria. The conventional thinking is that it will cause human traffic problem if an embassy attracts protesters/sit-ins, while in another country that I visited the embassy area is the most well protected area so one might have to look twice as hard at whether it adds/subtracts points.

A river is definitely a no-no. No 60 years flood history doesn’t mean it will never happen. (http://www.itnews.com.au/news/qld-data-centres-operating-despite-floods-244399)s

 

Whose definition of Green Field

Now, the funniest thing about green field is the land owner or would-be land owner definition of green field. Three years ago, an intermediary who is a good friend was told that a piece of land in Zhejiang is good, it has pros of government approval should be easy and not too far from shanghai (2 hours was mentioned). When I asked about the actual address the day prior to the visit they cannot give the full details and my appreciation of Chinese folks of “not too far” was not honed by then yet. All I had was a future look picture that comprises condominium blocks surrounding a lake and ten of 6-storeys office blocks further behind those blocks. They said this is a conceptual layout, any of the office block can be changed into a data center building if the group I represent is interested.

Set off at 08:30am, we reached the place at 11:15am after circling and calling the county vice governor in charge of investment met us and our car followed him for 20 minutes. So it is nearly 3 hours from Shanghai bund in light traffic. I cannot imagine what sort of travel time during peak hours.

But the biggest surprise was that the land in question, is that the land is not cleared. Meaning, a village with 6 houses full of people sits on that land, ducks, chickens, ploughed vegetable plots and all.

The roads as shown on the conceptual picture is not even approved yet.

A good day was wasted, I reached back the city at 4pm.

It is what I would say, pre-cleared green field. From that point on, I insisted of someone else sending me a photo of the land in question.

Second story. The land in question has been selected by the owner because of all the good points for it to build factory building, worker quarters, warehouse, office blocks, and three data center buildings as it is the right land size and at a good rate. I have told them bluntly why choose that place to have a data center. But, the decision to house three data center buildings within the same plot of land is by the top man, we just have to make it work. Top man says, government approved, utility power and water supply both approved by government, and the government will give grant to offset the initial investment into tri-generation power and cooling plant. The bad? It has two big minuses. First, a river outside along the fence only mere meters away. It has a train track which is no more than 50 meters away with a train passing by every 10 minutes during peak hours. AND the three data center buildings has to be placed along that edge of fence as decided by the power-that-be! So, resistance is futile. But sales is futile as well. Nothing has been sold for 15 months and not much will be by the looks of things.

Tell it as it is, unsuitable. And yet they went ahead. After that when they asked what can they do to improve their chances, advice such as using the river water as backup cooled water is rejected because city has strengthened waterway protection.

 

Brown Field

A IT service company said that they are doing so well that they want to consolidate and move all their co-location customer who they park at other 3rd party data center to one that they want to build. But, I don’t know why but they are fixated about wanting to convert part of an exhibition complex to house data center. The exhibition halls will still be retained and use for big exhibition and events.

No matter how I told them that the high human traffic is not good, they just won’t listen. And there is no place to have backup generators and external fuel storage tanks given the high human traffic.

A few years ago, an industrial property broker brought me to visit an industrial building that is only 10% occupied. She said this building is good, high floor loading, high floor to ceiling height. But there are three big minuses, one is the largest floor plate is such that fire escape of other unit will have to pass through this space to reach the fire escape. So unless you lease the entire floor, you will have the cut through your space with fire escape corridor and let others through your protected data center space! Second it is next to an army base. Last but not least, helicopter flies overhead now and then. Got to check your surroundings. Visit the space more than once.

Actually there is an even worse example than the above shared industrial building type. Warehouse buildings are good as a choice to convert to data center building. BUT only if you have the entire building. There are two data center buildings I know that share the building with existing warehousing type tenant as the building owner cannot terminate the lease with the existing tenant. So the fire risk and high cargo truck traffic is detrimental to the data center businesses that operates in these facilities. It took years for the warehousing tenant lease to run out and the data center operator quickly outbid and got those spaces. I really don’t advice sharing space when come to warehouse type, get the entire space.

Nearly Ready Site

This type of site are plenty across China especially in tier 2 and tier 3 cities.

A building has been built, nearby completed but halted due to some financial or other problem.

This type of site to me, is the worst of them all. Last year (2015), I was brought to a provincial capital city in the western part of China. Three separate data center buildings, two were completed, and one has been in near completion state for 18 months. The company that I work for wants that nearly completed one for reasons not shared with me.

This near completed data center is very close to finishing in terms of physical building itself, with one wall on the chiller plant room opened to move in chilled-water chillers. However, one chiller sit half-in and half-out, so it was under rain and in the open for full 18 months. Most chilled water chiller type are indoor unit and this one is no exception.

One way we counter this risk is to ask the original owner of these chillers (fortunately the government agency that owns the building) to get the manufacturer to re-test these chillers and give warranty.

At the data center halls, raised floor are up on 20% of them, electrical cabling are not completed and we cannot assess how much was done and since they were not fully completed, we will not be sure if they will all work together if the remaining works are completed.

This type is the worst. Half done, untested, with no knowing if all the materials that are supposed to be in are all there after 18 months. It will cost more to put in the rest.

And as we engaged a quantity surveyor to check through the design, the delivered M&E equipment, cablings, while more unknowns are known but the list of unknowns will still be worrying to us. This evaluation project took 6 months and it still leaves me uncomfortable to proceed any further. I have said early on in this site evaluation that it be better to choose a green field in that city and I am not wrong 6 months later after spending money and time.

 

Existing Data Center

This one took place in Australia. An old data center building in west Sydney is looking for new owner. But the full story was not shared on why they are selling it. Upon visiting the site, we learnt that the bank that owns it will decommission this 25 years old site when their newest and latest is ready in a couple of months. Everything is old there. To take on new customer, new owner will need to completely retrofit it and put in all brand new M&E equipment. It is directly opposite a hardware superstore, which has lots of woods, paints, basically flammable stuff. It raises the fire risk for this site. This hardware superstore is about 10 years old when this street was more quiet and tranquil.

Beside the fire risk of its neighbor across the road, I think there is no good reason to buy a very old site without any tenant. It will be more cost effective to develop a brown site or green field.

Some Basic Lessons Learnt

You can get a full checklist by searching for one online and adapting / add with your organisation’s experience and key concerns.

Just a few points:

  1. Be inquisitive
  2. Ask around
  3. Take pictures
  4. Ask for detailed information – address, current owner, tenancy
  5. Authorities approval for land, building, power, water
  6. Look at who are the neighbors, is their business a source of fire risk, is there a potential competitor looking to build nearby
  7. It pays to get someone local to get the key information

 

Sharing on Data center site selection or evaluation

Blast from the past – a lesson on documentation

Background

I have joined one of the largest IT outsourcing company in Singapore which has most of the government IT outsourcing business as a technical service manager for the national public library that has headquarter, main branch and 9 large branches, total 1000 odd users. When I came on board, I learnt that I have a dozen staff and besides 2 IT engineers, and an almost non-existent helpdesk, the rest of the IT support staff are such that one IT support engineer supports two branches and I have no idea who is where. We gave such bad IT support service that I was surprised that we are still holding on to this account. The user IT director took back the network planning and support functions, and he is revamping the entire library IT network.

The account has been with our company for at the longest time, at least 8 years, but there is no documentation of LAN or the entire WAN network. Our company did not take good care of this account, no technical service manager lasted more than a year, mind you in was in the days of transition from main-frame / mini-computer to client server and LAN technology. All my predecessors do not like to take out this account which has implemented LAN/WAN and is planning a new LAN/WAN coinciding with a move of the library IT system cum computer room move to another site in the next three months. So this senior IT engineer, i.e. me, suddenly became the pseudo technical service manager even though officially there is a technical service manager but he is on the transition out to another account which needs his mainframe knowledge.

Entire Library System grinds into a crawl

In the middle of the day, suddenly frantic calls came in and one of my senior IT support engineer took the calls that told her the library books borrowing and returning system is extremely slow. I was next to the computer room which housed the mini-computer that runs that critical IT system for main library and all the branches. As the clock moved minute by minute, the system is almost non-accessible outside of the computer room local area network.

All of us are scratching our heads, and my supervisor calls in other senior IT support engineers but they are not able to find the problem. The user IT director gave us stern warnings both verbal and via email. At the end of the day by the time the libraries had closed, the problem still persisted.

The next day, the same problem continues to wreck havoc to the main function of books borrowing and returning and it was switched to manual borrowing and returning of books, a process that was not used after the automated IT system was used three to four years back.

My rudimentary knowledge of the main IT system (runs on Unix) and the computer room LAN and the wider WAN that links all the branches to the main library hub (the computer room is in the same main library building) and I being the man-in-charge (no one else has any clues and I am the manager on site) meant I took the brunt of the users and IT director’s fire. I ran ping to all the main system’s IP address and to all the branches’ network. A trend shows that out of all the time-outs only a few packets get through. Clearing running non-essential processes on the main IT system doesn’t seem to help much. By Friday’s evening, my wife came to meet me to have dinner at 8pm because I told her I would be there through-out the night to try and solve this problem. My staff Cheng Woon was there with me until 11+pm and I persisted.

I only knew than that the entire library LAN and WAN is one entire LAN, all the branches are bridged onto the main library LAN via bridge router that were never set to router mode, all the while its simply set as bridge mode; for reason never known.

Suspects

By midnight that night, I reason to myself, that it must be a network problem caused by an errant source that blast the entire main library LAN and WAN with network packets has flooded the entire main library LAN and WAN. I cannot easily find where is the source of the problem. Making the main IT system less loaded doesn’t help. I have to find out the source of the network storm, i.e. cut the bad branch to give the rest a chance to access the main IT system.

I came to the desperate move at about 3-4am on Saturday to pull out network cable one-by-one from the 5 to 6 LAN switches to find out if I can isolate the source. i.e. a cut and slash. Slowly but surely, when I split the network in halves, and then halves again, the main IT system became usually for the isolated LAN (at that time, we have not used VLAN, everything is one big LAN). But pulling out all the network cables still showed nearly the same symptoms, the problem is still there. That leads me to think, it has to be in the main library LAN but I have put out all the LAN FTP cables, what is left?

At about 8am, time for breakfast and coffee, I had my favorite wanton noodles from the old library coffee shop next to the side entrance of the old library building (it has since torn down in 2000s).

Eureka

I go back and start anew, now looking for the elusive LAN cable. I looked at a old IT rack cabinet, it has a couple of old LAN switches and running with flashing lights.

Curious, I peered around that IT rack, going behind the rack and found a couple of thick Ethernet cables coming out of the back of those two old LAN switches. Thick Ethernet cables were like used in the late 1980s. No ones use them anymore even by 1990s. But they were there, probably been there since 1980s. So, I unplugged them, praying for luck, lo and behold the network problem is gone. All my pings to the main IT system and all the branches suddenly back to normal condition. No dropped packets and responses are all within 100msec. Wow. I hung around for another 30 minutes and no change. So I called Cheng Woon to come in to replace me, and I went home to shower and take a rest. The user IT director was informed of this and he said not to change the manual book borrowing and returning process during this weekend and let things stayed as it is until Monday.

The Apparent Culprit

Everyone is in on Monday first thing in the morning, tested the system access without problem. And the system performed as per normal. In the afternoon, I was told by the user IT director that the thick Ethernet cable was traced to the basement level of the main library, where some of the main library support staff works while some had already been relocated to some of the branch libraries ahead of the closure of the main library building that is planned to take place a few months later. So a subcontractor that carries out removal works had cut some of the cables above the ceiling board. The thick Ethernet cable is supposed to be non-essential since it only serves to that section of the building linking to old PC connection (the details I do not know and it does not matter anyway), and if cut properly will not cause any issue. But the thick Ethernet cable works in a way that if cut incorrectly, the cutter caused two copper wires to touch each other that in turn causes the LAN switch to experience a transmit/receive error which gets broadcasted to the entire main library LAN, and that is the cause of the problem that plagued the main library system for three days. Just a non-essential cable that was supposed to be removed but cut in a certain way to cripple the entire organisation’s main IT system and the entire network for two days (library stays open on weekends in Singapore).

Tens of Thousands of public library patrons were forced to queue behind counters manned by staff who were recalled back to man those counter while previously they breeze through the automated self check-out systems. Complaints were published in newspapers and letters were written in.

I am glad that the library staff are very understanding, but to this day, first thing when I see any operations is to see if their documentation is comprehensive and updated.

from this incident, I learnt a few things:

  1. If we have documented the library LAN/WAN, we may be able to identify the problem much earlier and easier.
  2. If we have experienced staff who had been with the account, and knows the LAN/WAN and the systems, we could have save some time learning what we should have known. I think this is why the user IT director was not happy with us, but he did not blame me since I was only on the job for a month.
  3. If we have known where everything is and what they are for in the computer room, we may have found the problem much earlier.

 

The silver linings of this incident, and the subsequent problems that I managed in that site for that year, gained the confidence and trust of the user IT director and the CIO, that I have done several things right:

  1. Leave no stone unturned. Dig to the bottom, go back to basics.
  2. Take responsibility, your people and your users trust someone that take charge.
  3. Something small can trip something big, the small stuff that we ignore will cause big problem.images-of-singapore-in-60s-70s-6-728-national-library.jpg
Blast from the past – a lesson on documentation

A view on the China Data Center Market – part 1

Published 3rd March 2016

This will be the first post on a series of posts by me and a colleague on the China Data Center market. To be precise, China here refers to mainland China.

 

  1. Overall

As the opening post of this series on the China Data Center Market (“China DC Market”), we’ll provide an overview and key attributes of the China Data Center market. We’ll also discuss a pertinent issue which is the “IDC” license that confuses or causes concern to a would-be data center player considering to entering the China DC Market.

  1. Purpose

This series of posts is meant to help a foreign data center stakeholder who is looking beyond the material of a research report of market size, statistics and such. It is not meant to be comprehensive; it is meant to be sharing of experience.

  1. China Data Center landscape

Just like when the term Asia Pacific Data Center Market is used, it is used as a generalization to a certain extent but there are key markets within the Asia Pacific Data Center Market that are unique and different from each other. The same is with China.

In China, all of big Chinese companies classify China into at least three and sometimes up to seven regions for ease of managing their business operations, among them you will find that they center around major cities such as Beijing, Shanghai, GuangDong, Nanjing, ChongQing and ChengDu. Each region/city is unique and different from one another. There are local practices in the different cities that one will need to find out and follow given that the people and dominant players there are different.

A lot has been said about China, which itself is a sign of the growing importance of China. It has the most number of Internet users. The 2014 statistics shows that it has more than combined number 2 to 4 (US, India and Japan). 2015 total shows it has 688 million users. 71% of those Chinese users are mobile Internet users.

Foreign data center service providers have been looking at the China DC market for a long time. Google is quietly picking up where it left off 5 years ago and heading back into China mainland.

The imbalance of the critical network resources and utility power supply means that data centers will need to be close to major network hubs while also it need be in an area where utility power capacity is available.

However, in another report, US leads China 4:1 in terms of major cloud and Internet data center sites, where China is number 2. I see things in positives, so I think China will have more room and therefore there is a lot of opportunities to be involved while China grows faster than the rest of the world.

  1. Definition

Just to be clear on what categories of data center are we talking about.

For the purpose of this series, we define the data center market to be layered (the foremost being the lowest stack):

categories-of-data-centers

The simpler three layers suffice for the purpose of this first post.

  1. Growth Drivers

We need to understand what is driving the growth of data center in China and how is it different from other markets.

As highlighted above, a large number of China’s Internet users are mobile users. They consume content, order goods and services (taxis, train tickets, airline tickets, hotels, fast food delivery, groceries, movie tickets) which is basically everything a person needs through mobile app and mobile payment. They work using the mobile chat app Wechat (including Wechat group), and they socialize using mobile app like Wechat, QQ, MoMo, MeiTu, etc.

Internet and Mobile Video content is very big in China. People watch videos on the train, bus, in office. Second is mobile games.

The banking industry is relatively slower than their counterparts in tying in more financial and personal services onto the mobile platform. Wechat does the equivalent for the China’s Internet users with their Wechat pay.

For China, it is the mobile users that are driving the big Internet data growth, as they work, consume, pay, and play through their mobile apps.

The popular acronym BAT refers to Baidu, Alibaba, and Tencent. BAT is a big factor for taking up a lot of the data center space in the last 5 years in China, fueled by the growth in volume of work/consume/pay/play of mobile users.

  1. Key Government Entities for Data Center

Central tax authority (federal tax)

Ministry of Industry and Information Technology (IDC licensing)

Local government (for land lease, local tax, business registration)

Power authority (for utility power)

Telcos (for carrier access)
More will be elaborated in subsequent posts during this series. Interesting to note is that there is no concept of free hold land in China. You lease the land from the State for certain x number of years to build your data center facility.

Another important piece of information is that the power supply company and the grid is usually one and the same, but they may or may not “listen” to the local government. So make sure you secure BOTH the land and the required utility power capacity and power lines before you commit to any significant investment.

  1. China Data Center Players

The well known data center companies are the dominant telcos: China Telecom (“CT”), China Unicom (“CU”), and China Mobile (“CM”) controls between 75-80% of the market. CT and CU are the big boys in the China DC Market with CM a distant and trying very hard to catch up third place.

The carrier neutral players that this author knows are are 21vianet (listed on Nasdaq), GDS, DrPeng (Shanghai stock exchange), Inspur, Centrin, Shanghai Data Solutions.

There are lots of data center buildings sprinkled throughout China. For Tier-2 or Tier-3 cities (i.e. at municipality or county level, which is one or two rungs below a provincial capital),  Quite a fair number are owned by the municipality or county directly or through investment company controlled by the municipality or county.

For tier-1 cities, i.e. the big cities and provincial capitals, there are good data centers owned and managed by local government controlled investment companies usually as part of a bigger technological or high tech industrial park (like 25 sq km or bigger scale type of industrial park) that serves the companies in them. For example, the Suzhou International Park Data Center is a Uptime Institute Tier-4 certified data center.

For tier-1 cities, and for some tier-2 cities, a sizable proportion of data center facilities are owned and operated by individual companies with the following characteristics:

  • data center is not their main business, it is either in response to demand from their managed IT/ISP/ICP business or because they are owned a complex with 6-20 buildings and they decided to build/convert one of the building to a data center facility
  • data center is one of their main business and they do well locally where they are, but they don’t intend to grow beyond their industrial park/district/county/city

So there are probably hundreds of data center service providers running one or two data center facilities in China’s many cities. For example, Shanghai city issued 26+ IDC licenses. However, well known data center service providers in Shanghai is only a handful – GDS, Shanghai Data Solutions.

It has been rumored that Tencent sublease some of their data center space but this author cannot confirms it.

  1. Data Center Outsourcing is low, but will likely pick up pace

Most large Chinese companies be it state owned or private, despite IT infrastructure not being their core business, will want to build their own data center facility.

However, there is good potential for subsidiaries or branch companies of large Chinese companies, and medium Chinese companies which are still substantial in size to outsource their data center colocation requirements.

Another reason is the spillover, which is also substantial.

Baidu, Alibaba, and Tencent will on first instinct build their data centers, it normally couldn’t build them fast enough and don’t have the right land in the right place to meet those demands. They will turn to the dominant telcos first to lease the data center space.

Large Chinese enterprises has been and continues to build their own data centers. Wanda Group, owned by the richest man in China in 2015 – Mr Wang Jianlin, builds a O2O cloud data center complex in a county of ChengDu city that will house more than 3,000 IT racks of medium power density.

Last but not least, cloud technology has made outsourcing of IT infrastructure service more acceptable and it is fueling a strong growth in the data center colocation business. AWS has entered the China market in 2013, and Aliyun lead a pack of 30+ public and private cloud service providers in a hot expansion pace in China (Aliyun is the cloud product name of Alibaba).

  1. Internet Data Center (“IDC”) license

Caveat: this is not a lawyer speak nor can you rely on for your critical business decision making.

IDC is under a subcategory called value added services of the telecommunications act which is controlled by the Ministry of Industry and Information Technology. Foreign company, cannot owns more than 50% of a company that operates under the IDC license. While Pacnet had announced back in 2013 (http://www.marketwired.com/press-release/pacnet-awarded-idc-license-miit-provide-data-center-network-services-chongqing-china-1843849.htm) that they are the first and only foreign invested company to obtain an IDC license, but dig deeper you will find the name Pacnet Business Solutions which is a 50-50 joint venture with a Chinese state-owned enterprise. This is the only way and the maximum is 50% exact.

But, wait a minute, a lot of wholesale colocation data center service provider do not provide, or do not from day 1 need to, provide the Internet connectivity service.  Then, you have two choices, waive and not provide and thus go it through a 100% owned local China subsidiary, or still partner will someone locally to help with the sales pipeline and the retail colocation agreement and provide that last mile connectivity. The choice is yours. Clear and simple, right?

It was not so clear mainly because the China Data Center market do not have anyone that only do wholesale colocation data center service! Why? Everyone wants to owns the value chain and provides everything all the way from basic rack up to servers and up to managed IT services.

So when government officials hear a foreign data center service provider wants to come in, they are usually cool to the idea until you explain at length with each and everyone you meet (and you will meet a lot of Chinese officials) that you are only interested in the non-Internet-connectivity part of the IDC business.

The advice here is this, get a local Chinese partner who does the ISP/IDC portion, explain to the key people in the partner until they are all clear, then you proceed with this partner down this “IDC” route. Or you can lease the powered space to this partner and let him be the “IDC” licensee, then you can stay at arms-length from this “IDC” license issue altogether. It is one of the way what some of the foreign data center service provider that are already in China are doing.

If you still want to part owned the “IDC” license through a joint venture, remember you are limited up to 50%. There is another issue. IDC license can be issued nationwide, or at each city. But, the nationwide one still has to be registered locally in each city you intend to own a data center facility. Complicated? There is no choice but to comply. That attracts a fees and the local department of MIIT will random pick a local data center for audit. So it saves some communications and liaison difficulty if you get a local partner that is familiar with the IDC/ISP business.

What’s to come in the series on the China Data Center Market

The rest of the series, not necessarily in the order shown, will look at the following topics:

  1. Part two of the view of the overall China data center market
  2. Current China Data Center Market players
  3. Beijing Data Center market
  4. Shanghai Data Center market
  5. GuangZhou and ShenZhen Data Center market
  6. ChongQing+ChengDu Data Center market
  7. and other China cities that is considered promising growth markets
  8. Technical Advancement in China Data Center Market

 

Reference (some of these links are in Chinese):

  1. https://www.zhihu.com/question/20062534
  2. http://www.most.gov.cn/tztg/201206/W020120621537448430735.pdf
  3. https://www.cnnic.net.cn/hlwfzyj/hlwxzbg/201601/P020160122469130059846.pdf
  4. http://www.internetlivestats.com/internet-users-by-country/
  5. http://time.com/4023367/google-china/
  6. http://www2.itif.org/2014-china-ict.pdf
  7. http://www.datacenterdynamics.com/colo-cloud-/colocations-eastern-front/94890.article
  8. http://www.datacenterdynamics.com/design-build/dcd-as-a-service-theres-no-data-center-market-like-china/95115.article
  9. https://blog.equinix.com/2012/05/the-data-center-industry-in-hong-kong-and-greater-china/
  10. http://www.360doc.cn/article/15115512_381221644.html
  11. https://qiaodahai.com/internet-backbone-provider-ibp.html
  12. http://www.equinix.sg/company/news-and-events/press-releases/equinix-completes-acquisition-of-asia-tone-expands-data-center-footprint-in-asia-pacific/

…to be continued in part 2

 

A view on the China Data Center Market – part 1