Deciphering the slow web tech behind Philippines’ largest mobile networks

The Philippines Internet market is divided between the two whales: Globe Telecom and Smart Communications. Each has a share of over 60 million subscribers. Being a frequent Philippines traveller, I thought I should give a technical overview on the web technologies used by the two companies’s primary websites.

The security and performance attitude used by Philippines’ internet giants can clearly showcase why there’s need for a fix. A call to action is pending for Globe and Smart. And if they don’t listen – a new player in the Philippines telecommunications market will arrive to cut a big chunk of customers from them. A player with decent, performance oriented approach towards its websites.

Globe

How Globe uses DNS

Globe uses Cloudflare and custom DNS nameservers. The domain’s nameservers are Globe’s own. The www points to www.globe.com.ph.cdn.cloudflare.net via a CNAME record and 10 minutes TTL. The result is an extra DNS lookup, and little to no DNS caching.

The speed of Globe’s own DNS nameservers vs Cloudflare’s is most obvious from the screenshots:

The setup looks like something that would happen if Cloudflare was activated via a web control panel like Plesk or similar. (By the way, did you know how web panels are evil?).

The proper setup would involve:

• Cloudflare should be set as primary DNS provider for all DNS records. This will ensure speed, considering existing Cloudflare’s point of presence in Manila. Subdomains can be delegated to custom Globe’s nameservers only when absolutely needed, via NS records.
• The CNAME record for www should be an A record with a very high TTL value to efficiently cache the DNS record. How fast do you really change servers, Globe?

The ugly DNS setup resulted in more findings about Globe’s web infrastructure:

• globe.com.ph resolves to 54.241.3.103 (Amazon)
• www.globe.com.ph resolves to a Cloudflare IP (served via Cloudflare servers)
• web.globe.com.ph goes to 203.177.154.228 which is inside Globe’s own network in Makati

Primary globe.com.ph website

The main website www.globe.com.ph does not enforce the use of secure website. So we have a security and a SEO duplication issues.
It is hosted by an Amazon AWS server in ap-southeast-1 availability zone, which is located in Singapore. Considering Globe to be an internet giant of an island country, it seems severely bad choice to host their primary website elsewhere, not to mention off-shore. The increased latency is the result of this choice. Even if www is served by Cloudflare, the Cloudflare servers still have to talk back to Singapore based server, and provided that the website’s HTML is not cached in Cloudflare, this will be the case most of the time.

The connection between Cloudflare and the Amazon server is likely not secure. Globe’s Amazon server does not enforce SSL redirect when accessed over insecure HTTP protocol (Show direct access headers, Show Cloudflare access headers).

curl -IL -H "Accept-Encoding: br, gzip" -H "Host: www.globe.com.ph" 54.241.3.103
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=C6461E38B5B133F63F56B21C9E4E2C9E.4; Path=/
Content-Encoding: gzip
Liferay-Portal: Liferay Portal Community Edition 6.0.6 CE (Bunyan / Build 6006 / February 17, 2011)
ETag: "f25aca1"
Content-Type: text/html;charset=UTF-8
Content-Length: 20
Date: Thu, 19 Jul 2018 11:07:57 GMT
Connection: close
Server: Apache Tomcat/Liferay
Set-Cookie: SERVERNO=A; path=/
Cache-control: private

curl -IL -H "Accept-Encoding: br, gzip" https://www.globe.com.ph/
HTTP/2 200
date: Thu, 19 Jul 2018 14:46:33 GMT
content-type: text/html;charset=UTF-8
set-cookie: __cfduid=d3a8e04bd9254704218d4bd7c9bfc6eac1532011592; expires=Fri, 19-Jul-19 14:46:32 GMT; path=/; domain=.globe.com.ph; HttpOnly
set-cookie: JSESSIONID=512DD6E2A74BCB4F1457D6BAB364D0B2.4; Path=/; Secure
liferay-portal: Liferay Portal Community Edition 6.0.6 CE (Bunyan / Build 6006 / February 17, 2011)
set-cookie: SERVERNO=C; path=/
cache-control: private
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 43cdfa62fa01871b-ARN
content-encoding: gzip

The connection to the site is not fully secure due to mixed SSL content issue (see screenshot):

The primary website only supports HTTP/2 protocol only when served by Cloudflare, not directly.

The site is sending Cache-control: private header which makes public caching by proxies troublesome. Even though the homepage has no dynamic content whatsoever (at least absent active user session), it does not send any Expires headers, making browser caching troublesome as well. The Cache-control: private is required due to bad customer experience workflow (see below).

globe.com.ph software

The primary globe.com.ph website advertises the used software through HTTP headers. It uses open source product “Liferay Portal” which is Java based.

Liferay-Portal: Liferay Portal Community Edition 6.0.6 CE (Bunyan / Build 6006 / February 17, 2011)

It is year 2018, but Globe’s primary website software version is from Feb 2011. This is very old. Sending this header is a clear negligence of security and unnecessary bits of information.
Also as you might have guessed – Java itself is a memory resource hog. It requires servers with enough memory to allocate things as Java heaps.

Java applications require a virtual machine to run because they cannot be are usually not compiled directly into machine language (native code) for many processors, instead Java is usually complied into bytecode. Java is also a very high-level Object-Oriented programming language (OOP) which means that while the application code itself is much easier to maintain, the objects that are instantiated will use that much more memory.

The whole Liferay-Portal portal thing seems to rely on “portlets” which are separate apps on different subdomains. Another subdomain, another DNS lookup and performance suffers.

Globe’s My Account

accounts.globe.com.ph

It comes out that the sole purpose of accounts.globe.com.ph, from my testing, is just for displaying the login page. The accounts.globe.com.ph uses outdated nginx 1.4.7 with the stable being 1.14.0 at the time of writing. (Show headers)

HTTP/1.1 200 OK
Server: nginx/1.4.7
Date: Wed, 18 Jul 2018 19:49:08 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 35989
Connection: keep-alive
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache
Cache-Control: no-store
Set-Cookie: JSESSIONID=20C0E6696B8ABAFF166B6223FF628891; Path=/; HttpOnly

The whole issue with accounts.globe.com.ph is that for the end customer like myself, you’d only get to this domain for login purpose. An extra DNS lookup, SSL handshake occur only for the purpose of showing a login page. That’s pretty “amazing” on its own. If you setup a dedicated “site” for login, at least take care of secure TLS ciphers, won’t you?

Serious usability issues

I have to mention about the worst usability coupled with the worst security approach to the matter. Supposedly I want to visit my account, I enter globe.com.ph in a browser address bar. Here is the request flow that follows:

• You are redirected to http://www.globe.com.ph (to enforce www prefix). The redirect does not enforce SSL!
• Now you are at insecure version of the website: http://www.globe.com.ph
• Click Login to My Globe Account blue button
• You’re taken to https://accounts.globe.com.ph/login?service=http%3A%2F%2Fwww.globe.com.ph%2Fc%2Fportal%2Flogin
• Entering valid credentials takes you back to insecure homepage of http://www.globe.com.ph/:

So the whole thing is flawed: you click a login on one “website”, taken to another domain just for login, then back to the homepage that presents nothing but a handful of action links. Of course, the proper workflow involves seeing your account details right after login, with supplementary actions offered as other menu items.

If you’re lucky to go through login nightmare workflow and viewing your account details now, you’re finally at myaccount.globe.com.ph, which is powered by Apache/2.4.6 (Red Hat Enterprise Linux) Communique/4.2.2. At last, that’s where TLS encryption is being enforced (Show headers)

curl -IL -H "Accept-Encoding: br, gzip" myaccount.globe.com.ph
HTTP/1.1 302 Found
Cache-control: no-cache="set-cookie"
Content-Type: text/html; charset=iso-8859-1
Date: Thu, 19 Jul 2018 11:16:32 GMT
Location: https://myaccount.globe.com.ph/
Server: Apache/2.4.6 (Red Hat Enterprise Linux) Communique/4.2.2
Set-Cookie: AWSELB=85E38F2914CA1F16AADFB7AB83358965779ADE5D2C2B9D5431AEC96B5DFF5399801F47E4C8760180CB4AA03E8DE7DC6B93C5096B995BC653CC2B687B67F1138ABF1B2125A7;PATH=/;MAX-AGE=900
X-Frame-Options: SAMEORIGIN
Connection: keep-alive

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-control: no-cache="set-cookie"
Content-Encoding: gzip
Content-Length: 17694
Content-Type: text/html; charset=UTF-8
Date: Thu, 19 Jul 2018 11:16:33 GMT
ETag: "17a04-571574fa0325f-gzip"
Last-Modified: Thu, 19 Jul 2018 10:04:55 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) Communique/4.2.2
Set-Cookie: AWSELB=85E38F2914CA1F16AADFB7AB83358965779ADE5D2C2B9D5431AEC96B5DFF5399801F47E4C82AF44346B87FD35546BD694887FFDFA75BC653CC2B687B67F1138ABF1B2125A7;PATH=/;MAX-AGE=900
Vary: Accept-Encoding,User-Agent
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Connection: keep-alive

Conclusions about Globe web infrastructure:

• None of the underlying servers support HTTP/2 protocol natively
• None of the servers support Brotli compression
• Software version information is clearly advertised, which makes it easy to search for CVE information by interested hackers
• No EV SSL certificates anywhere
• Quite a jungle of different software in use shows a very disorganised approach to web development

Smart

How Smart uses DNS

Similar to Globe, it uses custom DNS nameservers. There is a total of 4 nameservers and one of them is not responding. It is advised to use a DNS provider who knows what they are doing. And Cloudflare would be a good choice for hosting domain DNS.

The TTL for A record is only 15 mins. It is of course required to raise the value to something very high to ensure DNS caching.

Smart’s main website ensures SSL, which already feels great compared to Globe (Show headers).

curl -IL -H "Accept-Encoding: br, gzip" http://smart.com.ph/
HTTP/1.1 301 MOVED PERMANENTLY
Location: http://smart.com.ph/corporate

HTTP/1.1 301 MOVED PERMANENTLY
Location:https://smart.com.ph/corporate

HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Expires: -1
Vary: Accept-Encoding
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 19 Jul 2018 11:22:24 GMT
Set-Cookie: NSC_Q_MC_TNBSU.DPN.QI_SPPU=ffffffff09a2efd845525d5f4f58455e445a4a423660;path=/;secure

However visiting the homepage triggers a redirect. Why not present something useful at the root of the website? A redirect in this case is an unnecessary performance hit. As you can see from the headers, there are actually 2 redirects.

It uses an EV SSL certificate, however the whole purpose of EV SSL (displaying company name, ensuring user trust) is nullified through mixed SSL content issue on the redirected homepage, same as Globe’s:

The unaffected pages display company name properly:

The server does not support keep-alive connections nor HTTP/2 protocol. And it’s using a Windows server… quite an exotic choice these days, which of course won’t support Brotli compression.

The smart.com.ph is hosted in Pasig City, which is good, compared to hosting off-shore done by Globe.

Online account management (My Smart)

It seems that Smart takes security strongly with their choice of protective measure in the online account management. Same as the primary company website, it ensures TLS encryption. And there are bunch of security headers implemented in order to prevent XSS attacks (Show headers)

curl -IL -H "Accept-Encoding: br, gzip" http://my.smart.com.ph/
HTTP/1.1 301 MOVED PERMANENTLY
Location:https://my.smart.com.ph/

HTTP/1.1 302 Found : Moved Temporarily
Location: https://my.smart.com.ph/loginAuth
Connection: close
Cache-Control: no-cache
Pragma: no-cache

HTTP/1.1 302 Found
Cache-Control: private
Content-Length: 194
Content-Type: text/html; charset=utf-8
Location: https://smart.com.ph/corporate/message/notfound.html?aspxerrorpath=/loginAuth
Vary: Accept-Encoding
X-WebKit-CSP: default-src 'self'
X-Content-Security-Policy: default-src *; script-src https://code.jquery.com https://smart.com.ph https://*.googleadservices.com https://*.googletagmanager.com https://*.smart.com.ph https://*.googleapis.com https://*.facebook.com https://*.fbcdn.net https://*.facebook.net *.google-analytics.com https://*.google-analytics.com *.virtualearth.net *.google.com 127.0.0.1:* https://googleads.g.doubleclick.net https://googleads.g.doubleclick.net/* https://*.googleads.g.doubleclick.net/*  https://api.usabilla.com https://pagead2.googlesyndication.com https://w.usabilla.com https://ad.doubleclick.net 'self' 'unsafe-inline' 'unsafe-eval';style-src https://*.cloudfront.net https://smart.com.ph https://*.googleapis.com https://*.smart.com.ph https://*.gstatic.com 'self' 'unsafe-inline';connect-src https://accounts.google.com https://www.facebook.com https://smart.com.ph https://*.smart.com.ph 'self';frame-ancestors https://smart.com.ph https://*.smart.com.ph 'self' http://foxplus.com http://www.foxplus.com;form-action 'self';img-src https://stats.g.doubleclick.net https://w.usabilla.com https://pagead2.googlesyndication.com https://*.cloudfront.net https://googleads4.g.doubleclick.net https://www.google.com https://www.google.com.ph https://smart.com.ph https://googleads.g.doubleclick.net https://googleads.g.doubleclick.net/* https://*.googleads.g.doubleclick.net/* https://*.googleadservices.com https://*.googletagmanager.com https://*.smart.com.ph https://*.googleapis.com https://*.facebook.com 'self' data:;child-src http://*.facebook.com https://accounts.google.com https://*.facebook.com https://*.doubleclick.net https://smart.com.ph https://*.smart.com.ph 'self'
Content-Security-Policy: default-src *; script-src https://code.jquery.com https://smart.com.ph https://*.googleadservices.com https://*.googletagmanager.com https://*.smart.com.ph https://*.googleapis.com https://*.facebook.com https://*.fbcdn.net https://*.facebook.net *.google-analytics.com https://*.google-analytics.com *.virtualearth.net *.google.com 127.0.0.1:* https://googleads.g.doubleclick.net https://googleads.g.doubleclick.net/* https://*.googleads.g.doubleclick.net/*  https://api.usabilla.com https://pagead2.googlesyndication.com https://w.usabilla.com https://ad.doubleclick.net 'self' 'unsafe-inline' 'unsafe-eval';style-src https://*.cloudfront.net https://smart.com.ph https://*.googleapis.com https://*.smart.com.ph https://*.gstatic.com 'self' 'unsafe-inline';connect-src https://accounts.google.com https://www.facebook.com https://smart.com.ph https://*.smart.com.ph 'self';frame-ancestors https://smart.com.ph https://*.smart.com.ph 'self' http://foxplus.com http://www.foxplus.com;form-action 'self';img-src https://stats.g.doubleclick.net https://w.usabilla.com https://pagead2.googlesyndication.com https://*.cloudfront.net https://googleads4.g.doubleclick.net https://www.google.com https://www.google.com.ph https://smart.com.ph https://googleads.g.doubleclick.net https://googleads.g.doubleclick.net/* https://*.googleads.g.doubleclick.net/* https://*.googleadservices.com https://*.googletagmanager.com https://*.smart.com.ph https://*.googleapis.com https://*.facebook.com 'self' data:;child-src http://*.facebook.com https://accounts.google.com https://*.facebook.com https://*.doubleclick.net https://smart.com.ph https://*.smart.com.ph 'self'
X-Frame-Options: ALLOW FROM https://accounts.google.com/o/oauth2 https://smart.com.ph https://*.smart.com.ph http://foxplus.com http://www.foxplus.com
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-UA-Compatible: IE=edge
Date: Thu, 19 Jul 2018 11:48:42 GMT
Set-Cookie: NSC_Q_MC_NZ.TNBSU.DPN.QI_MPHJOBVUI=ffffffff09a2ef4545525d5f4f58455e445a4a422971;path=/;secure

HTTP/1.1 301 Moved Permanently
Cache-Control: private, max-age=86400
Content-Length: 149
Content-Type: text/html; charset=utf-8
Location: /Corporate/message/notfound.html
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
Date: Thu, 19 Jul 2018 11:48:43 GMT
Set-Cookie: NSC_Q_MC_TNBSU.DPN.QI_SPPU=ffffffff09a2efd845525d5f4f58455e445a4a423660;path=/;secure

HTTP/1.1 200 OK
Cache-Control: max-age=2678400
Content-Length: 4869
Content-Type: text/html
Content-Encoding: gzip
Last-Modified: Mon, 29 Jan 2018 10:40:00 GMT
Accept-Ranges: bytes
ETag: "0803783ed98d31:0"
Vary: Accept-Encoding
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
Date: Thu, 19 Jul 2018 11:48:43 GMT
Set-Cookie: NSC_Q_MC_TNBSU.DPN.QI_SPPU=ffffffff09a2efd845525d5f4f58455e445a4a423660;path=/;secure

At my.smart.com.ph you will see an EV SSL in use. However, the SSL score is B, so there are still some obvious security concerns about it:

Conclusions about Smart web infrastructure:

• None of the servers support HTTP/2 protocol
• None of the servers support Brotli compression
• Software version information is clearly advertised, which makes it easy to search for CVE information by interested hackers
• EV SSL certificates are implemented in both primary and personal account areas, however mixed SSL issue is negating the EV SSL benefits at primary website
• The software stack is more consistent across the servers

Wrapping it up

Globe neglects security at large. And both providers have taken little measure to address performance concerns.
None of the mentioned websites support HTTP/2 protocol. This protocol is supported by over 80% of Internet users and greatly reduces network latency through its request multiplexing.

None of the server maintainers have bothered to implement Brotli compression to reduce websites’ payload.

Much of the slow Internet experience in the Philippines is contributed by inconsistent software policies within the two mobile giants. And Globe is leading in the “inconsistency” competition.

While the above analysis doesn’t correspond to technical quality of either mobile network – the performance and security issues outlined above can serve as a clear representation of each company’s internet tech etiquette and attitude.

Each website tested has the following problems:

• No OSCP stapling, making SSL connections slower, because clients have to validate certificates revocation themselves
• Lack HTTP/2 protocol support
• Lack of Brotli compression
• TLS certificates unnecessarily bear anchor certificate, resulting in extra unnecessary payload during website visits
• DNS caching should be improved through setting much higher TTL values
• Custom DNS nameservers are of no good use
• No or little protective measures against information disclosure for server software.

Both companies are largely non-innovative when it comes to web software use. They spend billions expanding their actual mobile networks, yet the simple obvious measures in regards with their customer facing websites are not being taken.

The Philippines internet market is indeed divided. Between too whales. And perhaps the insights above will give you an idea why with these kind approaches and negligence, the Internet in Philippines is expensive and crawling.. Crawling to a halt.