What is SOX Compliance?

The SOX Act protections require companies to maintain a thorough, accurate knowledge of their financial data and upkeep their network security in all areas where financial data could be breached or misrepresented. Passed by the U.S. Congress in 2002 after several major fraud cases, including the Enron fraud scandal, SOX guards investors against faulty or misrepresented disclosures of publicly traded companies’ financial data.

At a high level, SOX mandates that companies do the following:

• Prepare complete financial reports to ensure the integrity of financial reporting and regulatory compliance
• Put controls in place to safeguard financial data and ensure its accuracy
• Provide year-end financial disclosure reports
• Protect employee whistleblowers who disclose fraud

SOX also requires CEOs, CFOs, and other C-suite executives to take responsibility for honest financial data reporting, formalized data security policies and procedures, and documentation of all relevant financial details—which can all be pulled up and reviewed via audit at any time. But SOX also puts pressure on IT teams, much like other government, regulatory agency, and jurisdictional compliance policies like the European Union’s General Data Protection Regulation (GDPR), through its data and reporting requirements.

Data-Specific Rules in SOX

SOX specifically regulates the financial data of publicly traded companies, especially as it relates to corporate transactions, which can include line items like off-balance sheet transactions, pro forma figures, and stock transactions. The law enacts several rules for these kinds of financial data, obliging companies to submit for regular external audits and enabling internal reporting and controls to support financial data accuracy.

Data management and archiving are essential to SOX. IT must create and maintain a data archive of corporate records that conforms to the management of electronic records provisions of SOX Section 802, which provide direction in three critical areas:

• Retention periods for records storage are defined, as are SOX best practices for the secure storage of all business records
• Definitions must be made for the various types of business records that need to be stored (e.g.,  business records, communications, electronic communications, etc.)
• Guidelines must be in place for the destruction, alteration, or falsification of records and the resulting penalties

Beyond routine audits and maintenance of financial reporting, companies are expected to report concrete evidence of changes in financial condition to the SEC. The controls that SOX requires include an Internal Control Report, which details all financial history for managerial responsibility and transparency, as well as additional documentation that proves the regular monitoring of financial data.

The SEC also requires formal data security policies with proof of communication and enforcement across a corporate network. SOX does not provide exact security protocols or expectations.

SOX Compliance

SOX compliance falls into the category of corporate governance and accountability. While it’s mainly financial, it also involves enterprise IT departments as it includes very specific guidelines for how corporate electronic records must be stored and for how long—generally, for a minimum period of five years.

SOX directs all covered companies to undergo annual audits and make the results publicly available to their stakeholders. In order to pass a compliance audit for SOX, companies need to inspect the quality of their internal controls and systems in these four key areas:

• Limiting physical and electronic access to only what authorized users absolutely need
• Security measures with features like endpoint security, multi-factor authentication, and anti-malware have been set up and maintained to protect against breaches
• Secure backup storage for all relevant financial data that could suffer from a breach
• Change management and internal auditing practices to ensure financial data remains protected when users, devices, and programs change
• Appropriate reporting cycles, report formats, and data content must be put into place with a documented review process for SOX reports

SOX Enforcement

Externally, SOX is enforced by the U.S. Securities and Exchange Commission, which established the Public Company Accounting Oversight Board to oversee, regulate, and discipline auditors who work with publicly traded companies under SOX.

All publicly traded companies with American shareholders are required to comply with SOX rules, including related boards, management, and accounting firms. The consequences for non-compliance can be fines, imprisonment or both.

SOX can also be applied to international companies in certain situations—like other data laws, such as GDPR, SOX applies to any publicly traded company that does business with American citizens, even if the business itself is not located in the United States.

SOX Benefits

SOX has ushered in a level of financial accountability and liability that makes it difficult for publicly traded companies to defraud or mismanage financials. It has improved corporate data governance and ethics and made financial responsibility both a management and a board-level mandate. SOX also delivers a number of additional benefits for IT.

More Widespread Acceptance

Traditionally, management has not always recognized the return on investment of IT projects, but SOX has changed that to some extent. For example, it may be easier to approve the purchase of data integration and cleaning software, additional data storage, or expensive security and activity monitoring software if it’s necessary to help the company stay SOX compliant. Similarly, IT policies that might have been viewed as unnecessary or ignored because they might delay project deliverables now must be documented for compliance.

Reduced Complexity

SOX forces the integration of systems, work processes, and data that might not otherwise be integrated. Many companies use multiple invoicing, purchasing, enterprise resource planning (ERP), and customer relationship management (CRM) systems that IT needs to support. To maintain compliance with SOX, those systems are more likely to be integrated and business processes and systems redesigned to make everything—including data—more seamless and uniform. This integration reduces system complexity for IT in both new application development and system maintenance.

Supplier Data Sharing

SOX can improve the quality of transactions and data sharing with suppliers. While IT has traditionally struggled to integrate internal systems with those of suppliers for data exchange, SOX elevates the issue of supplier data incompatibilities into a SOX narrative for uniform data standards. This can compel supplier audits and demands for change to better integrate supplier data with corporate systems.

Improved Data Quality

The need to conform to external regulatory requirements has placed the spotlight on clean and accurate data and reporting and highlighted the importance of high quality data—even if it means investing IT staff time and budget. Standardized, high quality data is now the goal of virtually every company; without it, it’s almost impossible to run analytic and automation technologies like artificial intelligence. SOX and other compliance regulations help facilitate this work.

SOX Challenges

Despite the benefits of compliance—not least of which is avoiding punishment and fines—companies face challenges in their ongoing efforts to meet SOX regulations, which can put burdens on multiple departments and teams. Here are some of the most common.

Lack of Expertise

Inadequate resources or internal SOX expertise can be a problem for many companies, especially new and/or smaller businesses. Compliance requires implementing appropriate controls to monitor each SOX-related process—for example, purchasing might implement a control so that only someone manager-level or higher can sign off on an order in excess of $1,000. If the existing purchasing system does not have that checkpoint built into it, unsigned invoices could slip through and create a material weakness for auditors or regulators to find.

Company Culture

Some company cultures are averse to having rules and regulations foisted upon them. For example, some  technology startups pride themselves on creativity, freedom, and innovation—such environments make it difficult to get management onboard with costly, time-consuming, and restrictive SOX initiatives.

Data Integration 

Just because SOX requires data integration and uniform data management doesn’t make the job of data integration any easier for IT—it will take time, money, and resources. Businesses that merge or go through acquisitions and subsequently have to blend disparate systems into a composite enterprise whole for SOX reporting, especially, may find the effort daunting.

Regulatory Changes

The regulatory environment is constantly changing, and companies need to keep up. When a SOX requirement changes, the typical chain of communication starts in a regulatory agency, trickles down to the legal staff, gets reviewed by management, and then finally makes its way to IT. The challenge comes in keeping delays from happening along the way so that IT has time to implement the changes before the deadline.

The Bottom Line: SOX Compliance and Enterprise IT

SOX compliance is a fact of life for publicly traded companies. IT plays a major role in assuring that SOX guidelines and requirements are met. While the burden is high—and so are the costs for not meeting it—the advantages of compliance are widespread and benefit the companies themselves, not just their investors. SOX compliance has also elevated the role of IT in enterprise businesses, giving it a seat at the table it did not necessarily have prior. As similar new data regulations start to take hold around the world, IT teams will continue to play an important role in helping businesses stay compliant.

Read about the future of data management to learn about how other trends and policies are shaping the way enterprise organizations work with data.

How Does Master Data Management Work?

As organizations continue to take in data on an unprecedented scale—and increasingly rely on that data to inform everything from decision-making and operations to customer relationships and business intelligence—their dependence upon that data grows. It needs to be accurate, consistent, and reliable.

Master data management describes the process of cleaning and preparing that data by deduplicating, reconciling, and enriching it before admitting it to a storage repository where it can be used and maintained. The purpose of advance data preparation and cleaning is to assure all users across the enterprise that the data is accurate and can be trusted.

This achieves two important goals:

• ensuring that business decisions and reporting are informed by accurate data
• reducing conflicts by giving everyone across the enterprise access to the same data

An organization’s master data records are referred to as golden records because the data they contain has been fastidiously cleaned, refined, and validated, representing the “best version of data truth.”

Master Data Management Processes

Master data management is a process that involves both people and technology, but it has to begin with organizational buy-in. Preparing and moving data to an MDM repository is tedious and can be expensive, and maintaining a single source of truth for an organization means establishing new ways of working with data to ensure that it remains accurate and consistent.

The first step involves identifying all relevant data sources and the “owners” who will be responsible for them. The data in these sources must all be reviewed. Depending on the size of the organization or how it has worked with data historically, this can be a time-consuming stage.

For example, consider an enterprise that has acquired another company that used entirely different systems. All data items from both sides must be cross-referenced for duplicate record types and reformatted to a uniform format. At the same time, it must be flagged for inaccuracies, anomalies, or incompleteness, and any imperfect records must be eliminated.

This painstaking work is usually performed with the help of data-mapping software, a tool often built-in to master data management systems. The IT team leading the MDM process then creates a structure for the master data records that maps these items to their names in the original source systems. Once the master data records are mapped to all variants in other systems, the next step is for the organization to decide how it wants to maintain and use data going forward.

One approach is to immediately consolidate all data to the uniform names in the MDM repository; another is to let users continue using the original, disparate names in their resident systems and let the master data management system automatically consolidate the data into the uniform data repository. Both approaches are valid and will depend upon the preferred workflow.

Benefits of Master Data Management

MDM benefits organizations in many ways, but here are some of the most common:

• Creates uniform data—every department across the organization uses the same golden record data, ensuring that it is consistent, accurate, and reliable.
• Assists with regulatory compliance—aggregate information from disparate departments and systems can be difficult to gather and can sometimes be in conflict, but standardized MDM data is in a single place and presents a more accurate picture.
• Reduces IT asset costs—eliminating duplicate, incomplete, and extraneous data reduces the amount of needed storage capacity and saves money on storage and processing hardware.
• Improves customer satisfaction—sales and service referencing identical data can lead to better outcomes by giving everyone who interacts with customers a 360-degree view of the customer experience.

An Example of Master Data

Master data is the data at the heart of company operations, and encompasses a wide range of information about people, places, transactions, communications, interactions, and other business operations.

As an example of how it might work, a company might track three types of master data in its MDM repository:

• End data items, such as “customer” or “product” information
• Stored transactional data, such as how much of a product the customer orders and on what dates
• Derived analytical data, such as the average order amount and order frequency for a customer

From an operational standpoint, these three types of master data give the company a way of tracking the customer, the product, and the customer’s product consumption rate, and of predicting analytically when the customer is most likely to reorder and at what product quantity. Having a reliable record of this data can help with everything from budgeting to stocking to predicting sales and provide useful information for marketing campaigns and promotions.

Master Data Management Use Cases

Most organizations can benefit from implementing a master data management strategy, but it’s especially well suited for certain applications.

Mergers and Acquisitions

When one business acquires or merges with another, they need to also merge their data. Often the data is kept in different systems and formats, with different terminology. Master data management can help them identify commonalities and reconcile differences using uniform conventions, providing a more holistic, seamless data record.

Customer Service and Satisfaction

MDM can help create a 360-degree view of customers and the customer experience by unifying information that flows in from sales, service, fulfillment, returns, and even manufacturing and product development. When all of this information is consolidated into an MDM repository, any department can see how the customer has interacted with the organization. This enables employees to improve customer satisfaction and build customer loyalty and revenue potential.

Product Engineering and Manufacturing

Consolidating separate parts catalogs in purchasing, engineering, and manufacturing in an MDM repository can preclude duplicate orders and alert purchasers to issues that may have emerged in other departments. This can prevent mistakes that can happen when engineering product designs don’t sync with manufacturing bills of material. A uniform parts database can also consolidate outside part numbers and references to the same part—for example, a military specification part number that must be mapped to an internal part number for the same part.

Compliance and Regulation

Compliance auditors and regulatory officials are increasingly demanding cross-departmental reports that integrate data from across an entire organization. An MDM strategy that standardizes data from disparate departmental systems can make this hybrid reporting easier, facilitating compliance and avoiding missteps.

Master Data Management Challenges

Despite the clear benefits of master data management, implementation is difficult and can be costly. Here are the biggest challenges enterprises face when tackling MDM.

Organizational Buy-In

It’s easy to commit to an MDM project, but difficult to get everyone to do their part on a daily basis. MDM is not a one-and-done solution—it requires an ongoing commitment to implement in the first place, and to maintain over time.

Complexity

Standardizing data from a diversity of data sources is not easy work. How do you know for sure that one data name in the accounting system means the same thing as another variant in manufacturing, for example? End users most familiar with the systems must make interpretations and agree on a single, uniform term for data item variants.

Data Standards

There are differences in how systems record and format data. Regulatory requirements can make things worse—for example, a company that functions in multiple countries may find that some countries require numerics to be expressed into more than two places to the right of a decimal point while others do not. Meeting reporting requirements might require different data formats to be maintained in different systems, adding to the complexities of MDM.

Unstructured Data

Unlike traditional system-of-records data, unstructured data—photos, videos, emails, and text messages, for example—does not come with data item labels. These objects must be hand-annotated by users, a time- and labor-intense effort.

Timeline

MDM is a data infrastructure project that involves people and systems across an entire organization. It takes time to implement, and results are not always immediately visible. Stakeholders might see the time, effort, and money being spent without being able to readily see what it is or how it will deliver business value.

Trends in Master Data Management

Master data management is not new, but the field is evolving as organizations find themselves more and more dependent on data for all aspects of their work. As MDM grows in popularity, here are the trends shaping the market:

• An exponential growth of Internet of Things (IoT) data that must be consolidated and brought under management alongside other data.
• Vast amounts of unstructured data that must be annotated and linked with core system data.
• More corporate initiatives for customer-centric companies with a 360-degree view of customer data.
• The introduction of more artificial intelligence and machine learning that works on centralized data to uncover market, business, and operational trends.
• The movement to omnichannel sales and service where customer activity can be conducted and holistically integrated via chat, phone, and brick and mortar solutions.

Bottom Line: Going Forward with Master Data Management

Initiating a master data management strategy is a colossal effort, which historically has limited the work to very large enterprises where cross-channel and cross-departmental data integration is essential. Smaller companies might lack the resources to launch major MDM projects, but they  still have the need.

Technology is evolving to keep pace with the demand. Many vendors—enterprise resource planning (ERP) and customer relationship management (CRM) suppliers, for example—are already incorporating MDM tools directly into their systems to put them in reach of smaller businesses.

In all cases, data and data channels are expanding, and MDM systems capable of addressing all types of data and unifying them into single data expressions will be indispensable data management assets for organizations of all sizes. The volume of data is only going to continue growing, and businesses need to plan for how they consistently store and access data now and in the future.

Read The 5 Stages of Data Lifecycle Management to learn more about how organizations collect, interact with, and store the information that fuels their work.

Experts predict 175 zettabytes of data worldwide within two years, much of it coming from IoT (Internet of Things) devices. Companies of all sizes should expect significant troves of data, most of it unstructured and not necessarily compatible with system of record (SOR) databases that have long driven mission-critical enterprise systems like enterprise resource planning (ERP).

Even unstructured data should be subject to many of the same rules that govern structured SOR data. For example, unstructured data must be secured with the highest levels of data integrity and reliability if the business is to depend on it. It must also meet regulatory and internal governance standards, and it must be able to move freely among systems and applications on clouds, internal data repositories, and mobile storage.

To keep pace with the enormous demands of managing voluminous high velocity and variegated data day-in and day-out, software-based tools and automation must be incorporated into data management practices. Newer automation technologies like data observability will only grow in importance, especially as user citizen development and localized data use expand.

All of these forces require careful consideration as enterprise IT builds its data management roadmap. Accordingly, here are seven emergent data management trends in 2023.

Hybrid End-to-End Data Management Frameworks

Enterprises can expect huge amounts of structured and unstructured data coming in from a wide range of sources, including outside cloud providers; IoT devices, robots, drones, RF readers, and MRI or CNC machines; internal SOR systems; and remote users working on smart phones and notepads. All of this data might be committed to long- or short- term storage in the on-premise data center, in a cloud, or on a mobile or distributed server platform. In some cases, real-time data may need to be monitored and/or accessed as it streams in real time.

In this hybrid environment, the data, its uses, and its users are diverse—data managers will need data management and security software that can span all of these hybrid activities and uses so data can be safely and securely transported and stored point to point.

IBM is a leader in the data management framework space, but SAP, Tibco, Talend, Oracle, and others also offer end-to end data fabric management solutions. A second aspect of data management is being able to secure data, no matter where it is sent from or where it resides—end-to-end security mesh software from vendors such as Fortinet, Palo Alto Networks, and Crowdstrike can meet this need.

The Consolidation of Data Observability Tools

Because many applications now use multiple cloud and on-premises platforms to access and process data, observability—the ability to track data and events across multiple platform and system barriers with software—is a key focus for enterprises looking to monitor end-to-end movements of data and applications. The issue with most organizations that are using observability tools today is that they are using too many different tools to effect end-to-end data and application visibility across platforms.

Vendors like Middleware and Datadog recognize this and are focused on delivering integrated, “single pane of glass” observability tool sets. These tools enable enterprises to reduce the number of different observability tools they use into a single toolset that’s able to monitor data and event movements across multiple cloud and on premises systems and platforms.

Master Data Management for Legacy Systems

As businesses move forward with new technologies, they face the challenge of figuring out what to do with older ones. But some of those continue to provide value as legacy systems—systems that are outdated or that continue to run mission-critical functions vital to the enterprise.

Some of these legacy systems—for example, enterprise resource planning (ERP) systems like SAP or Oracle—offer comprehensive, integrated master data management (MDM) toolsets for managing data on their cloud or on-premises solutions. Increasingly enterprises using these systems are adopting and deploying these MDM toolsets as part of their overall data governance strategies.

MDM tools offer user-friendly ways to manage system data and to import data from outside sources. MDM software provides a single view of the data, no matter where it resides, and IT sets the MDM business rules for data consistency, quality, security, and governance.

Data Management Using AI/ML

While the trend of using artificial intelligence and machine learning (AI/ML) for data management is not new, it continues to grow in popularity driven by big data concerns as the unprecedented volume of data enterprises are faced with managing collides with an ongoing staffing shortage across the tech industry as a whole—especially in data-focused roles.

AI and ML introduce highly valuable automation to manual processes that have been prone to human error. Foundational data management tasks like data identification and classification can be handled more efficiently and accurately by advanced technologies in the AI/ML space, and enterprises are using it to support more advanced data management tasks such as:

• Data cataloging
• Metadata management
• Data mapping
• Anomaly detection
• Metadata auto-discovery
• Data governance control monitoring

As AI/ML continues to evolve, we can expect to see software solutions that offer intelligent, learning-based approaches including search, discovery, and capacity planning.

Prioritizing Data Security

In the first quarter of 2023, over six million data records were breached worldwide. A data breach can destroy a company’s reputation, impact revenue, endanger customer loyalty, and get people fired.This is why security of all IT—especially as more IT moves to the edge and the IoT—is an important priority for CIOs and a major IT investment area.

To meet data security challenges, security solution providers are moving toward more end-to-end security fabric solutions. They are offering training for employees and IT, since increases in user citizen development and poor user security habits can be major causes of breaches.

Although many of these security functions will be performed by the IT and network groups, clean, secure, and reliable data is foremost a database administrator, data analyst, and data storage concern as well.

Automating Data Preparation

The exponential growth of big data volumes and a shrinking pool of data science talent is stressing organizations. In some cases, more than 60 percent of expensive data science time is spent cleaning and preparing data.

Software vendors want to change this corporate pain point with an increase in data preparation and cleaning automation software that can perform these tedious, manual operations. Automated data preparation solutions ingest, store, organize, and maintain data, often using AI and ML, and can handle such manually intensive tasks as data preparation and data cleansing.

Using Blockchain and Distributed Ledger Technology

Distributed ledger systems enable enterprises to maintain more secure transaction records, track assets, and keep audit trails. This technology, along with blockchain technology, stores data in a decentralized form that cannot be altered, improving the authenticity and accuracy of records related to data handling. This includes financial transaction data, sensitive data retrieval activity, and more.

Blockchain technology can be used in data management to improve the security, shareability, and consistency of data. It can also be used to provide automatic verification, offering avenues to improve data governance and security.

Bottom Line: The Future of Data Management

As businesses confront the need to collect and analyze massive volumes of data from a variety of sources, they seek new means of data management that can keep pace with the expanding need. Cutting edge technologies like AI/ML and blockchain can be used to automate and enhance some aspects of data management, and software vendors are incorporating them into their platforms to make them an integral part of the work. As new technologies continue to evolve, data management methods will evolve with them, integrating them into processes driven by increasing demand.

Read next: Structured Data: Examples, Sources, and How it Works