Databricks-Certified-Data-Engineer-Professional最も有効な質問と解答で勉強

A data engineer created a daily batch ingestion pipeline using a cluster with the latest DBR version to store banking transaction data, and persisted it in a MANAGED DELTA table called prod.gold.all_banking_transactions_daily. The data engineer is constantly receiving complaints from business users who query this table ad hoc through a SQL Serverless Warehouse about poor query performance. Upon analysis, the data engineer identified that these users frequently use high- cardinality columns as filters. The engineer now seeks to implement a data layout optimization technique that is incremental, easy to maintain, and can evolve over time. Which command should the data engineer implement?

• A. Alter the table to use Z-ORDER and implement a periodic OPTIMIZE command.
• B. Alter the table to use Liquid Clustering and implement a periodic OPTIMIZE command.
• C. Alter the table to use Hive-Style Partitions and implement a periodic OPTIMIZE command.
• D. Alter the table to use Hive-Style Partitions + Z-ORDER and implement a periodic OPTIMIZE command.

A data engineer is designing a system leveraging Lakeflow Declarative Pipeline technology to process real-time truck telemetry data ingested from JSON files in S3 using Auto Loader. The data includes truck_id, timestamp, location, speed, and fuel_level. The system must support two use cases:
- Near-real-time monitoring of the latest location, speed, and
fuel_level per truck_id for the operations team.
- Daily aggregated reports of total distance traveled and average fuel
efficiency per truck_id for the management team.
Which approach should the data engineer use for streaming tables and materialized views in the Lakeflow Declarative Pipeline to meet these requirements?

• A. Define a materialized view to ingest and store the raw telemetry data, and create a streaming table to compute the latest location, speed, and fuel_level per truck_id for real-time monitoring.Create another materialized view to compute the daily aggregated distance and fuel efficiency per truck_id for reporting.
• B. Define a streaming table to ingest and store the raw telemetry data, and create a materialized view to compute the latest location, speed, and fuel_level per truck_id for real-time monitoring.
  Create another materialized view to compute the daily aggregated distance and fuel efficiency per truck_id for reporting.
• C. Define a streaming table to ingest and store the raw telemetry data, and create a streaming table to compute the daily aggregated distance and fuel efficiency per truck_id reporting. Create a materialized view to compute the latest location, speed, and fuel_level per truck_id for real-time monitoring.
• D. Define a streaming table to ingest and store the raw telemetry data, and create a streaming table to incrementally compute the latest location, speed, and fuel_level per truck_id for real-time monitoring. Create a materialized view to compute the daily aggregated distance and fuel efficiency per truck_id for reporting.

The data governance team is reviewing code used for deleting records for compliance with GDPR. They note the following logic is used to delete records from the Delta Lake table named users.

Assuming that user_id is a unique identifying key and that delete_requests contains all users that have requested deletion, which statement describes whether successfully executing the above logic guarantees that the records to be deleted are no longer accessible and why?

• A. Yes; the Delta cache immediately updates to reflect the latest data files recorded to disk.
• B. Yes; Delta Lake ACID guarantees provide assurance that the delete command succeeded fully and permanently purged these records.
• C. No; the Delta Lake delete command only provides ACID guarantees when combined with the merge into command.
• D. No; files containing deleted records may still be accessible with time travel until a vacuum command is used to remove invalidated data files.
• E. No; the Delta cache may return records from previous versions of the table until the cluster is restarted.

Given the following PySpark code snippet in a Databricks notebook:
filtered_df = spark.read.format("delta").load("/mnt/data/large_table")
\
.filter("event_date > '2024-01-01'")
filtered_df.count()
The data engineer notices from the Query Profiler that the scan operator for filtered_df is reading almost all files, despite the filter being applied.
What is the probable reason for poor data skipping?

• A. The event_date column is outside the table's partitioning and Z-ordering scheme.
• B. The filter condition involves a data type excluded from data skipping support.
• C. The Delta table lacks optimization that enables dynamic file pruning.
• D. The filter is executed only after the full data scan, preventing data skipping.

A data engineering team uses Databricks Lakehouse Monitoring to track the percent_null metric for a critical column in their Delta table.
The profile metrics table (prod_catalog.prod_schema.customer_data_profile_metrics) stores hourly percent_null values.
The team wants to:
Trigger an alert when the daily average of percent_null exceeds 5% for
three consecutive days.
Ensure that notifications are not spammed during sustained issues.

• A. SELECT AVG(percent_null) AS daily_avg
  FROM prod_catalog.prod_schema.customer_data_profile_metrics
  WHERE window.end >= CURRENT_TIMESTAMP - INTERVAL '3' DAY
  Alert Condition: daily_avg > 5
  Notification Frequency: Each time alert is evaluated
• B. WITH daily_avg AS (
  SELECT DATE_TRUNC('DAY', window.end) AS day,
  AVG(percent_null) AS avg_null
  FROM prod_catalog.prod_schema.customer_data_profile_metrics
  GROUP BY DATE_TRUNC('DAY', window.end)
  )
  SELECT day, avg_null
  FROM daily_avg
  ORDER BY day DESC
  LIMIT 3
  Alert Condition: ALL avg_null > 5 for the latest 3 rows
  Notification Frequency: Just once
• C. SELECT percent_null
  FROM prod_catalog.prod_schema.customer_data_profile_metrics
  WHERE window.end >= CURRENT_TIMESTAMP - INTERVAL '1' DAY
  Alert Condition: percent_null > 5
  Notification Frequency: At most every 24 hours
• D. SELECT SUM(CASE WHEN percent_null > 5 THEN 1 ELSE 0 END) AS violation_days FROM prod_catalog.prod_schema.customer_data_profile_metrics WHERE window.end >= CURRENT_TIMESTAMP - INTERVAL '3' DAY Alert Condition: violation_days >= 3 Notification Frequency: Just once