Search results
Imperial Treasure Restaurant Group. Jan 2013 - Present 11 years 6 months. Singapore. -- · Experience: Imperial Treasure Restaurant Group · Location: Singapore · 33 connections on LinkedIn. View...
- Tak Shun Leung - Audit Manager - RSM Hong Kong | LinkedIn
About. I am the member of Hong Kong Institute of Certified...
- 3 "Tak Shun Leung" profiles | LinkedIn
View the profiles of professionals named "Tak Shun Leung" on...
- Tak Shun Leung - Audit Manager - RSM Hong Kong | LinkedIn
- ABSTRACT
- 1. INTRODUCTION
- 2.1 Assumptions
- 2.3 Architecture
- 2.4 Single Master
- GFS chunkserver
- 2.5 Chunk Size
- 2.6 Metadata
- 2.6.2 Chunk Locations
- 2.6.3 Operation Log
- 2.7 Consistency Model
- 2.7.2 Implications for Applications
- 3. SYSTEM INTERACTIONS
- 3.1 Leases and Mutation Order
- 3.3 Atomic Record Appends
- 3.4 Snapshot
- 4. MASTER OPERATION
- 4.2 Replica Placement
- 4.3 Creation, Re-replication, Rebalancing
- 4.4 Garbage Collection
- 4.4.1 Mechanism
- 4.4.2 Discussion
- 4.5 Stale Replica Detection
- 5. FAULT TOLERANCE AND DIAGNOSIS
- 5.1 High Availability
- 5.1.2 Chunk Replication
- 5.1.3 Master Replication
- 5.2 Data Integrity
- 5.3 Diagnostic Tools
- 6. MEASUREMENTS
- 6.3.1 Methodology and Caveats
- 6.3.4 Master Workload
- 7. EXPERIENCES
- 9. CONCLUSIONS
- ACKNOWLEDGMENTS
We have designed and implemented the Google File Sys-tem, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous dis-tri...
We have designed and implemented the Google File Sys-tem (GFS) to meet the rapidly growing demands of Google’s data processing needs. GFS shares many of the same goals as previous distributed file systems such as performance, scalability, reliability, and availability. However, its design has been driven by key observations of our application work-...
In designing a file system for our needs, we have been guided by assumptions that offer both challenges and op-portunities. We alluded to some key observations earlier and now lay out our assumptions in more details. The system is built from many inexpensive commodity components that often fail. It must constantly monitor itself and detect, tolerat...
A GFS cluster consists of a single master and multiple chunkservers and is accessed by multiple clients, as shown in Figure 1. Each of these is typically a commodity Linux machine running a user-level server process. It is easy to run both a chunkserver and a client on the same machine, as long as machine resources permit and the lower reliability ...
Having a single master vastly simplifies our design and enables the master to make sophisticated chunk placement Application
Linux file system GFS chunkserver Linux file system Data messages Control messages Figure 1: GFS Architecture and replication decisions using global knowledge. However, we must minimize its involvement in reads and writes so that it does not become a bottleneck. Clients never read and write file data through the master. Instead, a client asks the m...
Chunksize is one of the key design parameters. We have chosen 64 MB, which is much larger than typical file sys-tem blocksizes. Each chunkreplica is stored as a plain Linux file on a chunkserver and is extended only as needed. Lazy space allocation avoids wasting space due to internal fragmentation, perhaps the greatest objection against such a lar...
The master stores three major types of metadata: the file and chunknamespaces, the mapping from files to chunks, and the locations of each chunk’s replicas. All metadata is kept in the master’s memory. The first two types (names-paces and file-to-chunkmapping) are also kept persistent by logging mutations to an operation log stored on the mas-ter’s...
The master does not keep a persistent record of which chunkservers have a replica of a given chunk. It simply polls chunkservers for that information at startup. The master can keep itself up-to-date thereafter because it controls all chunkplacement and monitors chunkserver status with reg-ular HeartBeat messages. We initially attempted to keep chu...
The operation log contains a historical record of critical metadata changes. It is central to GFS. Not only is it the only persistent record of metadata, but it also serves as a logical time line that defines the order of concurrent op-erations. Files and chunks, as well as their versions (see Section 4.5), are all uniquely and eternally identified...
GFS has a relaxed consistency model that supports our highly distributed applications well but remains relatively simple and efficient to implement. We now discuss GFS’s guarantees and what they mean to applications. We also highlight how GFS maintains these guarantees but leave the details to other parts of the paper.
GFS applications can accommodate the relaxed consis-tency model with a few simple techniques already needed for other purposes: relying on appends rather than overwrites, checkpointing, and writing self-validating, self-identifying records. Practically all our applications mutate files by appending rather than overwriting. In one typical use, a wri...
We designed the system to minimize the master’s involve-ment in all operations. With that background, we now de-scribe how the client, master, and chunkservers interact to implement data mutations, atomic record append, and snap-shot.
A mutation is an operation that changes the contents or metadata of a chunksuch as a write or an append opera-tion. Each mutation is performed at all the chunk’s replicas. We use leases to maintain a consistent mutation order across replicas. The master grants a chunklease to one of the repli-cas, which we call the primary. The primary picks a seri...
GFS provides an atomic append operation called record append. In a traditional write, the client specifies the off-set at which data is to be written. Concurrent writes to the same region are not serializable: the region may end up containing data fragments from multiple clients. In a record append, however, the client specifies only the data. GFS ...
The snapshot operation makes a copy of a file or a direc-tory tree (the “source”) almost instantaneously, while min-imizing any interruptions of ongoing mutations. Our users use it to quickly create branch copies of huge data sets (and often copies of those copies, recursively), or to checkpoint the current state before experimenting with changes t...
The master executes all namespace operations. In addi-tion, it manages chunkreplicas throughout the system: it makes placement decisions, creates new chunks and hence replicas, and coordinates various system-wide activities to keep chunks fully replicated, to balance load across all the chunkservers, and to reclaim unused storage. We now dis-cuss e...
A GFS cluster is highly distributed at more levels than one. It typically has hundreds of chunkservers spread across many machine racks. These chunkservers in turn may be accessed from hundreds of clients from the same or different racks. Communication between two machines on different racks may cross one or more network switches. Addition-ally, ba...
Chunkreplicas are created for three reasons: chunkcre-ation, re-replication, and rebalancing. When the master creates a chunk, it chooses where to place the initially empty replicas. It considers several fac-tors. (1) We want to place new replicas on chunkservers with below-average diskspace utilization. Over time this will equalize diskutilization...
After a file is deleted, GFS does not immediately reclaim the available physical storage. It does so only lazily during regular garbage collection at both the file and chunklevels. We find that this approach makes the system much simpler and more reliable.
When a file is deleted by the application, the master logs the deletion immediately just like other changes. However instead of reclaiming resources immediately, the file is just renamed to a hidden name that includes the deletion times-tamp. During the master’s regular scan of the file system namespace, it removes any such hidden files if they hav...
Although distributed garbage collection is a hard problem that demands complicated solutions in the context of pro-gramming languages, it is quite simple in our case. We can easily identify all references to chunks: they are in the file-to-chunkmappings maintained exclusively by the master. We can also easily identify all the chunkreplicas: they ar...
Chunkreplicas may become stale if a chunkserver fails and misses mutations to the chunkwhile it is down. For each chunk, the master maintains a chunk version number to distinguish between up-to-date and stale replicas. Whenever the master grants a new lease on a chunk, it increases the chunkversion number and informs the up-to-date replicas. The ma...
One of our greatest challenges in designing the system is dealing with frequent component failures. The quality and quantity of components together make these problems more the norm than the exception: we cannot completely trust the machines, nor can we completely trust the disks. Com-ponent failures can result in an unavailable system or, worse, c...
Among hundreds of servers in a GFS cluster, some are bound to be unavailable at any given time. We keep the overall system highly available with two simple yet effective strategies: fast recovery and replication.
As discussed earlier, each chunkis replicated on multiple chunkservers on different racks. Users can specify different replication levels for different parts of the file namespace. The default is three. The master clones existing replicas as needed to keep each chunk fully replicated as chunkservers go offline or detect corrupted replicas through c...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
We wish to thankthe following people for their contributions to the system or the paper. Brain Bershad (our shepherd) and the anonymous reviewers gave us valuable comments and suggestions. Anurag Acharya, Jeff Dean, and David des-Jardins contributed to the early design. Fay Chang worked on comparison of replicas across chunkservers. Guy Ed-jlali wo...
- 269KB
- 15
Shun Tak Holdings Limited is a leading listed conglomerate with core businesses in property, transportation, hospitality and investment sectors. Established in 1972, the Company (HKSE 242) has been listed on the Hong Kong Stock Exchange since 1973.
Ms. Pansy Ho is the vice chairman, chief executive officer and a director of the board of Shun Tak – China Travel Shipping Investments Limited and the chairman of its executive committee, and is directly in charge of the Group’s transportation businesses, which include ferry and coach operations.
About. I am the member of Hong Kong Institute of Certified Public Accountants. I have been the manager of the RSM Hong Kong since December 2018 and has more than 10 years’...
- 16
- 18
- RSM Hong Kong
- Hong Kong SAR
View the profiles of professionals named "Tak Shun Leung" on LinkedIn. There are 3 professionals named "Tak Shun Leung", who use LinkedIn to exchange information, ideas, and...
Shun-Tak Leung. We have designed and implemented the Google File Sys- tem, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance...
