Incident Response in Google Cloud: Foundations

Key Takeaways

• Although Google Cloud is becoming more widely used, research and documentation surrounding incident response is limited, and for many aspects non-existent.
  
• Through multiple recent Google Cloud investigations, Sygnia’s research team has gained a profound understanding of its infrastructure and available forensic artifacts.
  
• Google Workspace and Google Cloud Platform share a strong link. In some scenarios, a compromised user account in one infrastructure might lead to a compromise in the second.
  
• In this blog series, Sygnia will explore incident response and threat hunting in GCP and Workspace, culminating with a webinar and the release of Sygnia’s Google Cloud forensic collection tool.

Introduction

Google Cloud Growing Popularity

Google Cloud offers customers the unique ability to use the same infrastructure responsible for Google’s products, like the Google search engine and YouTube. Growing faster than any other cloud service provider in 2022, Google has stated they are committed to evolving their cloud services to become more globally available and feature rich [1]. An increase in global business use has resulted in threat actors expanding their attempts to develop attack vectors and exploitation capabilities. Although Google Cloud is becoming more widely used, research and documentation surrounding incident response is limited, and for many aspects non-existent.

Introduction to Google Cloud

Google  Cloud is a suite of commercially available services consisting of public cloud infrastructure, productivity applications, identity and device management, application programming interfaces (APIs), and more. For many organizations, the heart of Google Cloud includes Google Workspace and Google Cloud Platform.

Google Workspace (formerly G Suite) offers organizations identity and device management, a flexible email solution (Gmail), and collaboration tools (Google Drive, Calendar, etc.).
Google Cloud Platform (GCP) provides public cloud infrastructure offering a wide breadth of computing services (compute, storage, networking, big data, and more).

While these Google services appear as distinct and separate solutions, Workspace and GCP share a strong link. If you are utilizing Workspace, then the infrastructure of GCP is present and by default, accessible for all users. However, if you are utilizing GCP only, Workspace is not present by default.

Sygnia’s Research – Google Cloud Forensics

In this blog series, we’ll explore incident response and threat hunting in GCP and Workspace. Additional articles examining forensic artifacts, threat hunting concepts, and analysis strategies, as well as introducing Sygnia’s Google Cloud forensic collection tool will be provided in the next blog publications.

This introductory blog post will cover foundational elements critical to scoping a Google Cloud environment. There are many forensic artifacts and principles that can reveal malicious activity in Google Cloud. We will start by examining a significant concept central to Google infrastructure, known as identity integration.

Understanding Identity Integration

What is Identity Integration in Google Cloud

Identity integration refers to the identity management solution used to provide authentication and authorization for Google Cloud services. When a company signs on to use Google Cloud, there must be a top-level container for all users, groups, configuration, and data [2]. A top-level container can either be a Workspace or Google Cloud Identity account. Although Workspace is often associated with productivity applications (Gmail, etc.), arguably more important is its identity management functionality for Google services. When considering their identity management functionality, Workspace and Google Cloud Identity are similar to traditional directory services (e.g., Active Directory and Azure Active Directory) in structure and application.

Google Cloud Identity (Cloud Identity) is a unified management system for identity, access, applications, and endpoints [3]. Cloud Identity is often used in scenarios where GCP is configured without Workspace. GCP’s Identity and Access Management (IAM) service may handle control and visibility for resources within, but not control over who can access GCP itself. This is where Cloud Identity comes into play.

Cybersecurity Considerations of Identity Integration

In the same way that examining the structure of an Active Directory environment can highlight security gaps, analyzing identity management for a Google Cloud ecosystem can reveal available forensic artifacts and unexpected privilege relationships. From a cybersecurity perspective, understanding service identity integration can help us answer the following questions:

1. Do different configurations of GCP, Workspace, and Cloud Identity affect available forensic evidence, and how should an organization or response team approach scoping and log collection during an incident response investigation?
2. By default, any Workspace user in the domain is granted access to GCP and given the Project Creator IAM role.
   1. The above is true even if GCP has not been configured or is not actively being used. If an organization uses Workspace but does not use GCP, can a threat actor laterally move to GCP and abuse its services?
   2. If an organization has a Google Workspace business email compromise (BEC), does this mean their existing GCP infrastructure is also compromised?
3. What default configurations exist that facilitate lateral movement, privilege escalation, and persistence across GCP and Workspace or Cloud Identity?

Three Types of Identity Integration

To answer the questions above, we first need to get acquainted with three basic terms associated with Google Cloud identity management:

• Service Identity: the identity management solution used to interface all Google Cloud products together.
  
• Identity Provider (IdP): the system that provides authentication services to applications within a federation and manages user identities.
  
• Service Provider: the entity providing the service being accessed through a federated relationship.
  

In addition, a table has been created to generalize and illustrate the primary types of identity integrations.

Google Cloud identity management allows Workspace, Cloud Identity, and GCP to be integrated together or utilized separately. As described in Figure 1, there are three primary integrations when considering these service identities:

1. The first integration describes scenarios where Google Workspace is used as the identity provider for GCP. This Workspace role occurs in addition to its productivity app utilities. This integration also covers scenarios where Workspace is being actively used while GCP is not.
   
2. The second integration describes scenarios where a company wants to use GCP for its computing services, but already has non-Google email and collaboration tools. Cloud Identity can be used as the sole IdP. When considering Workspace and Cloud Identity functionality, “for the purpose of managing users, groups, and authentication, the two products can largely be considered equivalent” [2].
   
3. The third integration shows that Cloud Identity and Workspace are capable of being an identity provider or service provider. If utilized only as a service provider, a company can integrate an alternative IdP to provision identities and perform authentication. Even if a company already has Okta, it cannot be used alone as an IdP to GCP – Okta must be integrated through Cloud Identity or Workspace.

As seen above, GCP with domain association requires the use of a Google service identity, which can either be (1) Google Cloud Identity or (2) Google Workspace. This is true even for scenarios where external IdP is integrated with Google Cloud.

Admin Console and API Access

Google Workspace and Cloud Identity use a common technical platform and therefore provide access to the same Admin console and API for management activity. Important identity and application-based forensic artifacts can be found in the Admin console or via API. Google documentation describes this technical platform relationship as sharing “the same set of APIs and administrative tools” [2]. The primary difference between the two are the available logs, reports, and application controls, which are discussed in the next section.

Identity integration and Incident Response

Now that we have discussed service identities and determined that forensic evidence may vary depending on identity integration, let us explore these concepts by answering the previously mentioned questions.

Google Cloud Scoping and Forensic Evidence Collection

Question: Do different configurations of GCP, Workspace, and Cloud Identity affect available forensic evidence, and how should an organization or response team approach scoping and log collection during a security incident?

Answer: Below is a high-level table that describes the forensic artifacts available to different identity integrations. Regardless of IdP integration, we gain access to identity-based logs, the management Directory, Admin reports, and the Alert Center, which are accessible through the Admin console or API. If an external IdP is being used, directly examining privilege relationships through the external platform may indicate whether a compromised account can access additional resources. A more detailed description of each forensic artifact will be covered in the next blog post.

Artifact Category	Forensic Artifact	Integration #1 (GCP with Workspace IdP)	Integration #2 (GCP with Cloud Identity IdP)	Integration #3 (GCP or Workspace with External IdP)
Identity (Workspace/ Cloud Identity/ External	Audit Logs	Y	Y	Y
	Admin Reports	Y	Y	Y
	Alert Center Alerts	Y	Y	Y
	Directory	Y	Y	Y
	External IdP Logs	N/A	N/A	Y
Application (Workspace)	Audit Logs (App-specific)	Y	N/A	Y*
	Admin Reports (App-specific)	Y	N/A	Y*
	Alert Center Alerts (App-specific)	Y	N/A	Y*
	Application Data	Y	N/A	Y*
Platform (GCP)	Logs	Y	Y	Y
	IAM Role Binding Structure	Y	Y	Y
	Asset Inventory Settings	Y	Y	Y
	Usage Metrics	Y	Y	Y
	Cloud Billing Reports	Y	Y	Y

Lateral movement from Google Workspace to GCP

Question: If an organization uses Google Workspace but does not use GCP, can a threat actor laterally move to GCP and abuse services?

Answer: By default, any Workspace user in the domain is granted access to GCP and given the Project Creator IAM role. At face value, it makes sense that a user should be able to create and exploit resources in GCP even if none exist.

However, in addition to the previously discussed privilege requirements and service restrictions established in the Admin console, a valid Cloud Billing account must be created separately from Workspace within GCP to use enabled products and services. For organizations that use Workspace but do not utilize GCP, a Cloud Billing account will not automatically be available for use. While the compromise of an account may yield the creation of limited free-tier projects, any further post-exploitation activity would require a threat actor to set up their own Cloud Billing account.  

Therefore, the opportunity for lateral movement into GCP from Workspace primarily occurs if GCP is being actively utilized and the compromised user has the necessary application and IAM access.

Question: If an organization has a Google Workspace business email compromise (BEC), does this mean their existing GCP infrastructure is also compromised?

Answer: This is dependent on existing GCP IAM permissions for the account that has been compromised. By default, any Workspace user in the domain is granted access to GCP and given the Project Creator IAM role. These settings do not imply that the user will automatically have access to all existing GCP resources. Unless the compromised user has been given explicit permissions within GCP through the IAM service, they will only be able to observe the parent organization and create limited free-tier projects. The exception to this rule is the accounts that have been assigned Super Admin role via the management Directory. Accounts with the Super Admin role have irrevocable Organization Administrator privileges, despite it not being directly visible in GCP’s IAM service. The discovery of this attack path requires an examination of the Workspace Directory and GCP IAM role binding’s structure.

Lateral Movement from GCP to Google Workspace

Question: What default or manual configurations exist that facilitate lateral movement, privilege escalation, and persistence across GCP and Workspace?

Answer: There are several possibilities for misconfigurations that facilitate a threat actor in performing malicious activities within a Google Cloud environment. One of the most impactful misconfigurations involves domain-wide delegation. A careless implementation of domain-wide delegation can allow a threat actor to laterally move from GCP to Workspace and gain control of the entire Google Cloud ecosystem.

Domain-wide delegation is a powerful feature that allows applications to access and interface with other users’ data across Workspace. For example, domain-wide delegation given to a migration app can sync user content from another service to Google Workspace. When domain-wide delegation is applied to an application, access is implemented through the assignment of individual “scopes”. When applied, these scopes interface the application with every user in the organization.

The main problem arises when an organization has enabled domain-wide delegation that allows unrestricted access to privileged users. Once the client (i.e., service account with domain-wide delegation) has been compromised, admin user privileges can be impersonated to gain control over an entire Google Cloud ecosystem.

Question: Which steps can an organization take to review and mitigate its current implementation of domain-wide delegation?

Answer: A generalized process for the manual examination of domain-wide delegation is as follows:

1. Gain access to Admin console
   
2. Navigate to Menu > Security > Access and data control > API controls
   
3. Under Domain-wide delegation, click “Manage domain-wide delegation”
   
4. Select a client and click “View details”
   
5. Map existing API clients and determine if they are necessary
   
6. Review an API client’s scopes and assert whether access to specified resources is required and/or presents high-risk
   
7. Repeat at a scheduled interval

Vision

Our intention with this series of blog posts is to arm the infosec community with the knowledge to rapidly scope and respond to security incidents in Google Cloud. Alongside Gartner’s forecasted growth of worldwide cloud services to reach nearly $600 billion in 2023 [4], Sygnia has observed a correlated growth in the number of incident response engagements involving public cloud infrastructure. Developing skills in cloud security is becoming more critical in responding to and remediating incidents that will inevitably include complex cloud environments. Stay tuned for additional blog posts diving into Google Cloud forensic artifacts and a webinar covering the release of Sygnia’s forensic collection tool!

References 

1. https://cloud.google.com/blog/topics/google-cloud-next/google-cloud-next22-wrap-up 
2. https://cloud.google.com/architecture/identity/overview-google-authentication#cloud_identity_or_g_suite_account
3. https://cloud.google.com/identity 
4. https://www.gartner.com/en/newsroom/press-releases/2022-04-19-gartner-forecasts-worldwide-public-cloud-end-user-spending-to-reach-nearly-500-billion-in-2022
    

Google Cloud forensics blog series

• Incident Response in Google Cloud: Foundations
• Incident Response in Google Cloud: Forensic Artifacts
• Announcing ‘Cirrus’ – New Opensource Tool to Combat Google Cloud Incident Response Challenges
• Proof of Concept: Overcoming Google Cloud Incident Response Issues with ‘Cirrus’

This advisory and any information or recommendation contained herein has been prepared for general informational purposes and is not intended to be used as a substitute for professional consultation on facts and circumstances specific to any entity. While we have made attempts to ensure the information contained herein has been obtained from reliable sources and to perform rigorous analysis, this advisory is based on initial rapid study, and needs to be treated accordingly. Sygnia is not responsible for any errors or omissions, or for the results obtained from the use of this Advisory. This Advisory is provided on an as-is basis, and without warranties of any kind.