[Q12-Q35] 2022 Updated Associate-Developer-Apache-Spark Tests Engine pdf - All Free Dumps Guaranteed!

2022 Updated Associate-Developer-Apache-Spark Tests Engine pdf - All Free Dumps Guaranteed!

Latest Databricks Certification Associate-Developer-Apache-Spark Actual Free Exam Questions

What is the salary of a Databricks Associate Developer Apache Spark Exam?

The Average salary of different countries for Databricks Associate Developer Apache Spark professionals:

• India - INR 3516457 per year

• UK - Pounds 33288 per year

• United States - USD 84,000 per year

 

NEW QUESTION 12
The code block displayed below contains an error. The code block should return the average of rows in column value grouped by unique storeId. Find the error.
Code block:
transactionsDf.agg("storeId").avg("value")

• A. agg should be replaced by groupBy.
• B. All column names should be wrapped in col() operators.
• C. Instead of avg("value"), avg(col("value")) should be used.
• D. The avg("value") should be specified as a second argument to agg() instead of being appended to it.
• E. "storeId" and "value" should be swapped.

Answer: A

Explanation:
Explanation
Static notebook | Dynamic notebook: See test 1
(https://flrs.github.io/spark_practice_tests_code/#1/30.html ,
https://bit.ly/sparkpracticeexams_import_instructions)

 

NEW QUESTION 13
In which order should the code blocks shown below be run in order to create a DataFrame that shows the mean of column predError of DataFrame transactionsDf per column storeId and productId, where productId should be either 2 or 3 and the returned DataFrame should be sorted in ascending order by column storeId, leaving out any nulls in that column?
DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+
2.|transactionId|predError|value|storeId|productId| f|
3.+-------------+---------+-----+-------+---------+----+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.| 4| null| null| 3| 2|null|
8.| 5| null| null| null| 2|null|
9.| 6| 3| 2| 25| 2|null|
10.+-------------+---------+-----+-------+---------+----+
1. .mean("predError")
2. .groupBy("storeId")
3. .orderBy("storeId")
4. transactionsDf.filter(transactionsDf.storeId.isNotNull())
5. .pivot("productId", [2, 3])

• A. 4, 1, 5, 2, 3
• B. 4, 2, 5, 1, 3
• C. 4, 3, 2, 5, 1
• D. 4, 5, 2, 3, 1
• E. 4, 2, 1

Answer: B

Explanation:
Explanation
Correct code block:
transactionsDf.filter(transactionsDf.storeId.isNotNull()).groupBy("storeId").pivot("productId", [2,
3]).mean("predError").orderBy("storeId")
Output of correct code block:
+-------+----+----+
|storeId| 2| 3|
+-------+----+----+
| 2| 6.0|null|
| 3|null|null|
| 25| 3.0| 3.0|
+-------+----+----+
This question is quite convoluted and requires you to think hard about the correct order of operations.
The pivot method also makes an appearance - a method that you may not know all that much about (yet).
At the first position in all answers is code block 4, so the question is essentially just about the ordering of the remaining 4 code blocks.
The question states that the returned DataFrame should be sorted by column storeId. So, it should make sense to have code block 3 which includes the orderBy operator at the very end of the code block. This leaves you with only two answer options.
Now, it is useful to know more about the context of pivot in PySpark. A common pattern is groupBy, pivot, and then another aggregating function, like mean. In the documentation linked below you can see that pivot is a method of pyspark.sql.GroupedData - meaning that before pivoting, you have to use groupBy. The only answer option matching this requirement is the one in which code block 2 (which includes groupBy) is stated before code block 5 (which includes pivot).
More info: pyspark.sql.GroupedData.pivot - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

 

NEW QUESTION 14
Which of the following options describes the responsibility of the executors in Spark?

• A. The executors accept jobs from the driver, analyze those jobs, and return results to the driver.
• B. The executors accept tasks from the driver, execute those tasks, and return results to the driver.
• C. The executors accept jobs from the driver, plan those jobs, and return results to the cluster manager.
• D. The executors accept tasks from the cluster manager, execute those tasks, and return results to the driver.
• E. The executors accept tasks from the driver, execute those tasks, and return results to the cluster manager.

Answer: B

Explanation:
Explanation
More info: Running Spark: an overview of Spark's runtime architecture - Manning (https://bit.ly/2RPmJn9)

 

NEW QUESTION 15
Which of the following code blocks returns only rows from DataFrame transactionsDf in which values in column productId are unique?

• A. transactionsDf.drop_duplicates(subset="productId")
• B. transactionsDf.distinct("productId")
• C. transactionsDf.dropDuplicates(subset=["productId"])
• D. transactionsDf.dropDuplicates(subset="productId")
• E. transactionsDf.unique("productId")

Answer: C

Explanation:
Explanation
Although the question suggests using a method called unique() here, that method does not actually exist in PySpark. In PySpark, it is called distinct(). But then, this method is not the right one to use here, since with distinct() we could filter out unique values in a specific column.
However, we want to return the entire rows here. So the trick is to use dropDuplicates with the subset keyword parameter. In the documentation for dropDuplicates, the examples show that subset should be used with a list. And this is exactly the key to solving this question: The productId column needs to be fed into the subset argument in a list, even though it is just a single column.
More info: pyspark.sql.DataFrame.dropDuplicates - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

 

NEW QUESTION 16
The code block displayed below contains an error. The code block is intended to join DataFrame itemsDf with the larger DataFrame transactionsDf on column itemId. Find the error.
Code block:
transactionsDf.join(itemsDf, "itemId", how="broadcast")

• A. The syntax is wrong, how= should be removed from the code block.
• B. The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.
• C. broadcast is not a valid join type.
• D. Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.
• E. The join method should be replaced by the broadcast method.

Answer: C

Explanation:
Explanation
broadcast is not a valid join type.
Correct! The code block should read transactionsDf.join(broadcast(itemsDf), "itemId"). This would imply an inner join (this is the default in DataFrame.join()), but since the join type is not given in the question, this would be a valid choice.
The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.
This option does not apply here, since the syntax around broadcasting is incorrect.
Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.
No, it is enabled by default, since the spark.sql.autoBroadcastJoinThreshold property is set to 10 MB by default. If that property would be set to -1, then broadcast joining would be disabled.
More info: Performance Tuning - Spark 3.1.1 Documentation (https://bit.ly/3gCz34r) The join method should be replaced by the broadcast method.
No, DataFrame has no broadcast() method.
The syntax is wrong, how= should be removed from the code block.
No, having the keyword argument how= is totally acceptable.

 

NEW QUESTION 17
Which of the following code blocks returns a 2-column DataFrame that shows the distinct values in column productId and the number of rows with that productId in DataFrame transactionsDf?

• A. transactionsDf.groupBy("productId").agg(col("value").count())
• B. transactionsDf.groupBy("productId").count()
• C. transactionsDf.count("productId")
• D. transactionsDf.groupBy("productId").select(count("value"))
• E. transactionsDf.count("productId").distinct()

Answer: B

Explanation:
Explanation
transactionsDf.groupBy("productId").count()
Correct. This code block first groups DataFrame transactionsDf by column productId and then counts the rows in each group.
transactionsDf.groupBy("productId").select(count("value"))
Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement.
transactionsDf.count("productId")
No. DataFrame.count() does not take any arguments.
transactionsDf.count("productId").distinct()
Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right.
transactionsDf.groupBy("productId").agg(col("value").count())
False. A Column object, as returned by col("value"), does not have a count() method. You can see all available methods for Column object linked in the Spark documentation below.
More info: pyspark.sql.DataFrame.count - PySpark 3.1.2 documentation, pyspark.sql.Column - PySpark
3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

 

NEW QUESTION 18
Which of the following code blocks produces the following output, given DataFrame transactionsDf?
Output:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- productId: integer (nullable = true)
7. |-- f: integer (nullable = true)
DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+
2.|transactionId|predError|value|storeId|productId| f|
3.+-------------+---------+-----+-------+---------+----+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.+-------------+---------+-----+-------+---------+----+

• A. transactionsDf.rdd.formatSchema()
• B. transactionsDf.rdd.printSchema()
• C. print(transactionsDf.schema)
• D. transactionsDf.schema.print()
• E. transactionsDf.printSchema()

Answer: E

Explanation:
Explanation
The output is the typical output of a DataFrame.printSchema() call. The DataFrame's RDD representation does not have a printSchema or formatSchema method (find available methods in the RDD documentation linked below). The output of print(transactionsDf.schema) is this:
StructType(List(StructField(transactionId,IntegerType,true),StructField(predError,IntegerType,true),StructField (value,IntegerType,true),StructField(storeId,IntegerType,true),StructField(productId,IntegerType,true),StructFiel It includes the same information as the nicely formatted original output, but is not nicely formatted itself. Lastly, the DataFrame's schema attribute does not have a print() method.
More info:
- pyspark.RDD: pyspark.RDD - PySpark 3.1.2 documentation
- DataFrame.printSchema(): pyspark.sql.DataFrame.printSchema - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2

 

NEW QUESTION 19
Which of the following statements about executors is correct?

• A. Executors are launched by the driver.
• B. Executors stop upon application completion by default.
• C. An executor can serve multiple applications.
• D. Each node hosts a single executor.
• E. Executors store data in memory only.

Answer: B

Explanation:
Explanation
Executors stop upon application completion by default.
Correct. Executors only persist during the lifetime of an application.
A notable exception to that is when Dynamic Resource Allocation is enabled (which it is not by default). With Dynamic Resource Allocation enabled, executors are terminated when they are idle, independent of whether the application has been completed or not.
An executor can serve multiple applications.
Wrong. An executor is always specific to the application. It is terminated when the application completes (exception see above).
Each node hosts a single executor.
No. Each node can host one or more executors.
Executors store data in memory only.
No. Executors can store data in memory or on disk.
Executors are launched by the driver.
Incorrect. Executors are launched by the cluster manager on behalf of the driver.
More info: Job Scheduling - Spark 3.1.2 Documentation, How Applications are Executed on a Spark Cluster | Anatomy of a Spark Application | InformIT, and Spark Jargon for Starters. This blog is to clear some of the... | by Mageswaran D | Medium

 

NEW QUESTION 20
Which of the following statements about storage levels is incorrect?

• A. DISK_ONLY will not use the worker node's memory.
• B. In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.
• C. MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.
• D. The cache operator on DataFrames is evaluated like a transformation.
• E. Caching can be undone using the DataFrame.unpersist() operator.

Answer: C

Explanation:
Explanation
MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.
Correct, this statement is wrong. Spark prioritizes storage in memory, and will only store data on disk that does not fit into memory.
DISK_ONLY will not use the worker node's memory.
Wrong, this statement is correct. DISK_ONLY keeps data only on the worker node's disk, but not in memory.
In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.
Wrong, this statement is correct. In fact, Spark does not have a provision to cache DataFrames in the driver (which sits on the edge node in client mode). Spark caches DataFrames in the executors' memory.
Caching can be undone using the DataFrame.unpersist() operator.
Wrong, this statement is correct. Caching, as achieved via the DataFrame.cache() or DataFrame.persist() operators can be undone using the DataFrame.unpersist() operator. This operator will remove all of its parts from the executors' memory and disk.
The cache operator on DataFrames is evaluated like a transformation.
Wrong, this statement is correct. DataFrame.cache() is evaluated like a transformation: Through lazy evaluation. This means that after calling DataFrame.cache() the command will not have any effect until you call a subsequent action, like DataFrame.cache().count().
More info: pyspark.sql.DataFrame.unpersist - PySpark 3.1.2 documentation

 

NEW QUESTION 21
The code block displayed below contains an error. The code block should return a DataFrame in which column predErrorAdded contains the results of Python function add_2_if_geq_3 as applied to numeric and nullable column predError in DataFrame transactionsDf. Find the error.
Code block:
1.def add_2_if_geq_3(x):
2. if x is None:
3. return x
4. elif x >= 3:
5. return x+2
6. return x
7.
8.add_2_if_geq_3_udf = udf(add_2_if_geq_3)
9.
10.transactionsDf.withColumnRenamed("predErrorAdded", add_2_if_geq_3_udf(col("predError")))

• A. The Python function is unable to handle null values, resulting in the code block crashing on execution.
• B. The udf() method does not declare a return type.
• C. The operator used to adding the column does not add column predErrorAdded to the DataFrame.
• D. Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.
• E. UDFs are only available through the SQL API, but not in the Python API as shown in the code block.

Answer: C

Explanation:
Explanation
Correct code block:
def add_2_if_geq_3(x):
if x is None:
return x
elif x >= 3:
return x+2
return x
add_2_if_geq_3_udf = udf(add_2_if_geq_3)
transactionsDf.withColumn("predErrorAdded", add_2_if_geq_3_udf(col("predError"))).show() Instead of withColumnRenamed, you should use the withColumn operator.
The udf() method does not declare a return type.
It is fine that the udf() method does not declare a return type, this is not a required argument. However, the default return type is StringType. This may not be the ideal return type for numeric, nullable data - but the code will run without specified return type nevertheless.
The Python function is unable to handle null values, resulting in the code block crashing on execution.
The Python function is able to handle null values, this is what the statement if x is None does.
UDFs are only available through the SQL API, but not in the Python API as shown in the code block.
No, they are available through the Python API. The code in the code block that concerns UDFs is correct.
Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.
You may choose to use the transactionsDf.predError syntax, but the col("predError") syntax is fine.

 

NEW QUESTION 22
Which of the following describes Spark's standalone deployment mode?

• A. Standalone mode means that the cluster does not contain the driver.
• B. Standalone mode uses a single JVM to run Spark driver and executor processes.
• C. Standalone mode is a viable solution for clusters that run multiple frameworks, not only Spark.
• D. Standalone mode uses only a single executor per worker per application.
• E. Standalone mode is how Spark runs on YARN and Mesos clusters.

Answer: D

Explanation:
Explanation
Standalone mode uses only a single executor per worker per application.
This is correct and a limitation of Spark's standalone mode.
Standalone mode is a viable solution for clusters that run multiple frameworks.
Incorrect. A limitation of standalone mode is that Apache Spark must be the only framework running on the cluster. If you would want to run multiple frameworks on the same cluster in parallel, for example Apache Spark and Apache Flink, you would consider the YARN deployment mode.
Standalone mode uses a single JVM to run Spark driver and executor processes.
No, this is what local mode does.
Standalone mode is how Spark runs on YARN and Mesos clusters.
No. YARN and Mesos modes are two deployment modes that are different from standalone mode. These modes allow Spark to run alongside other frameworks on a cluster. When Spark is run in standalone mode, only the Spark framework can run on the cluster.
Standalone mode means that the cluster does not contain the driver.
Incorrect, the cluster does not contain the driver in client mode, but in standalone mode the driver runs on a node in the cluster.
More info: Learning Spark, 2nd Edition, Chapter 1

 

NEW QUESTION 23
In which order should the code blocks shown below be run in order to assign articlesDf a DataFrame that lists all items in column attributes ordered by the number of times these items occur, from most to least often?
Sample of DataFrame articlesDf:
1.+------+-----------------------------+-------------------+
2.|itemId|attributes |supplier |
3.+------+-----------------------------+-------------------+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+------+-----------------------------+-------------------+

• A. 4, 5
• B. 1. articlesDf = articlesDf.groupby("col")
  2. articlesDf = articlesDf.select(explode(col("attributes")))
  3. articlesDf = articlesDf.orderBy("count").select("col")
  4. articlesDf = articlesDf.sort("count",ascending=False).select("col")
  5. articlesDf = articlesDf.groupby("col").count()
• C. 2, 5, 4
• D. 5, 2
• E. 2, 5, 3
• F. 2, 3, 4

Answer: F

Explanation:
Explanation
Correct code block:
articlesDf = articlesDf.select(explode(col('attributes')))
articlesDf = articlesDf.groupby('col').count()
articlesDf = articlesDf.sort('count',ascending=False).select('col')
Output of correct code block:
+-------+
| col|
+-------+
| summer|
| winter|
| blue|
| cozy|
| travel|
| fresh|
| red|
|cooling|
| green|
+-------+
Static notebook | Dynamic notebook: See test 2

 

NEW QUESTION 24
The code block displayed below contains at least one error. The code block should return a DataFrame with only one column, result. That column should include all values in column value from DataFrame transactionsDf raised to the power of 5, and a null value for rows in which there is no value in column value. Find the error(s).
Code block:
1.from pyspark.sql.functions import udf
2.from pyspark.sql import types as T
3.
4.transactionsDf.createOrReplaceTempView('transactions')
5.
6.def pow_5(x):
7. return x**5
8.
9.spark.udf.register(pow_5, 'power_5_udf', T.LongType())
10.spark.sql('SELECT power_5_udf(value) FROM transactions')

• A. The returned DataFrame includes multiple columns instead of just one column.
• B. The pow_5 method is unable to handle empty values in column value, the UDF function is not registered properly with the Spark driver, and the name of the column in the returned DataFrame is not result.
• C. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and the SparkSession cannot access the transactionsDf DataFrame.
• D. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and Spark driver does not call the UDF function appropriately.
• E. The pow_5 method is unable to handle empty values in column value and the name of the column in the returned DataFrame is not result.

Answer: D

Explanation:
Explanation
Correct code block:
from pyspark.sql.functions import udf
from pyspark.sql import types as T
transactionsDf.createOrReplaceTempView('transactions')
def pow_5(x):
if x:
return x**5
return x
spark.udf.register('power_5_udf', pow_5, T.LongType())
spark.sql('SELECT power_5_udf(value) AS result FROM transactions')
Here it is important to understand how the pow_5 method handles empty values. In the wrong code block above, the pow_5 method is unable to handle empty values and will throw an error, since Python's ** operator cannot deal with any null value Spark passes into method pow_5.
The order of arguments for registering the UDF function with Spark via spark.udf.register matters. In the code snippet in the question, the arguments for the SQL method name and the actual Python function are switched. You can read more about the arguments of spark.udf.register and see some examples of its usage in the documentation (link below).
Finally, you should recognize that in the original code block, an expression to rename column created through the UDF function is missing. The renaming is done by SQL's AS result argument.
Omitting that argument, you end up with the column name power_5_udf(value) and not result.
More info: pyspark.sql.functions.udf - PySpark 3.1.1 documentation

 

NEW QUESTION 25
Which of the following describes slots?

• A. Slots are dynamically created and destroyed in accordance with an executor's workload.
• B. A slot is always limited to a single core.
  Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.
• C. A Java Virtual Machine (JVM) working as an executor can be considered as a pool of slots for task execution.
• D. To optimize I/O performance, Spark stores data on disk in multiple slots.

Answer: C

Explanation:
Explanation
Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.
Wrong, executors communicate with the driver directly.
Slots are dynamically created and destroyed in accordance with an executor's workload.
No, Spark does not actively create and destroy slots in accordance with the workload. Per executor, slots are made available in accordance with how many cores per executor (property spark.executor.cores) and how many CPUs per task (property spark.task.cpus) the Spark configuration calls for.
A slot is always limited to a single core.
No, a slot can span multiple cores. If a task would require multiple cores, it would have to be executed through a slot that spans multiple cores.
In Spark documentation, "core" is often used interchangeably with "thread", although "thread" is the more accurate word. A single physical core may be able to make multiple threads available. So, it is better to say that a slot can span multiple threads.
To optimize I/O performance, Spark stores data on disk in multiple slots.
No - Spark stores data on disk in multiple partitions, not slots.
More info: Spark Architecture | Distributed Systems Architecture

 

NEW QUESTION 26
The code block displayed below contains an error. The code block is intended to write DataFrame transactionsDf to disk as a parquet file in location /FileStore/transactions_split, using column storeId as key for partitioning. Find the error.
Code block:
transactionsDf.write.format("parquet").partitionOn("storeId").save("/FileStore/transactions_split")A.

• A. Partitioning data by storeId is possible with the bucketBy expression, so partitionOn should be replaced by bucketBy.
• B. The format("parquet") expression should be removed and instead, the information should be added to the write expression like so: write("parquet").
• C. Partitioning data by storeId is possible with the partitionBy expression, so partitionOn should be replaced by partitionBy.
• D. The format("parquet") expression is inappropriate to use here, "parquet" should be passed as first argument to the save() operator and "/FileStore/transactions_split" as the second argument.
• E. partitionOn("storeId") should be called before the write operation.

Answer: C

Explanation:
Explanation
Correct code block:
transactionsDf.write.format("parquet").partitionBy("storeId").save("/FileStore/transactions_split") More info: partition by - Reading files which are written using PartitionBy or BucketBy in Spark - Stack Overflow Static notebook | Dynamic notebook: See test 1

 

NEW QUESTION 27
Which of the following statements about the differences between actions and transformations is correct?

• A. Actions are evaluated lazily, while transformations are not evaluated lazily.
• B. Actions generate RDDs, while transformations do not.
• C. Actions can trigger Adaptive Query Execution, while transformation cannot.
• D. Actions can be queued for delayed execution, while transformations can only be processed immediately.
• E. Actions do not send results to the driver, while transformations do.

Answer: C

Explanation:
Explanation
Actions can trigger Adaptive Query Execution, while transformation cannot.
Correct. Adaptive Query Execution optimizes queries at runtime. Since transformations are evaluated lazily, Spark does not have any runtime information to optimize the query until an action is called. If Adaptive Query Execution is enabled, Spark will then try to optimize the query based on the feedback it gathers while it is evaluating the query.
Actions can be queued for delayed execution, while transformations can only be processed immediately.
No, there is no such concept as "delayed execution" in Spark. Actions cannot be evaluated lazily, meaning that they are executed immediately.
Actions are evaluated lazily, while transformations are not evaluated lazily.
Incorrect, it is the other way around: Transformations are evaluated lazily and actions trigger their evaluation.
Actions generate RDDs, while transformations do not.
No. Transformations change the data and, since RDDs are immutable, generate new RDDs along the way.
Actions produce outputs in Python and data types (integers, lists, text files,...) based on the RDDs, but they do not generate them.
Here is a great tip on how to differentiate actions from transformations: If an operation returns a DataFrame, Dataset, or an RDD, it is a transformation. Otherwise, it is an action.
Actions do not send results to the driver, while transformations do.
No. Actions send results to the driver. Think about running DataFrame.count(). The result of this command will return a number to the driver. Transformations, however, do not send results back to the driver. They produce RDDs that remain on the worker nodes.
More info: What is the difference between a transformation and an action in Apache Spark? | Bartosz Mikulski, How to Speed up SQL Queries with Adaptive Query Execution

 

NEW QUESTION 28
The code block shown below should write DataFrame transactionsDf to disk at path csvPath as a single CSV file, using tabs (\t characters) as separators between columns, expressing missing values as string n/a, and omitting a header row with column names. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__.write.__2__(__3__, " ").__4__.__5__(csvPath)

• A. 1. csv
  2. option
  3. "sep"
  4. option("emptyValue", "n/a")
  5. path
  * 1. repartition(1)
  2. mode
  3. "sep"
  4. mode("nullValue", "n/a")
  5. csv
• B. 1. repartition(1)
  2. option
  3. "sep"
  4. option("nullValue", "n/a")
  5. csv
  (Correct)
• C. 1. coalesce(1)
  2. option
  3. "sep"
  4. option("header", True)
  5. path
• D. 1. coalesce(1)
  2. option
  3. "colsep"
  4. option("nullValue", "n/a")
  5. path

Answer: B

Explanation:
Explanation
Correct code block:
transactionsDf.repartition(1).write.option("sep", "\t").option("nullValue", "n/a").csv(csvPath) It is important here to understand that the question specifically asks for writing the DataFrame as a single CSV file. This should trigger you to think about partitions. By default, every partition is written as a separate file, so you need to include repatition(1) into your call. coalesce(1) works here, too!
Secondly, the question is very much an invitation to search through the parameters in the Spark documentation that work with DataFrameWriter.csv (link below). You will also need to know that you need an option() statement to apply these parameters.
The final concern is about the general call structure. Once you have called accessed write of your DataFrame, options follow and then you write the DataFrame with csv. Instead of csv(csvPath), you could also use save(csvPath, format='csv') here.
More info: pyspark.sql.DataFrameWriter.csv - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

 

NEW QUESTION 29
The code block displayed below contains an error. The code block should arrange the rows of DataFrame transactionsDf using information from two columns in an ordered fashion, arranging first by column value, showing smaller numbers at the top and greater numbers at the bottom, and then by column predError, for which all values should be arranged in the inverse way of the order of items in column value. Find the error.
Code block:
transactionsDf.orderBy('value', asc_nulls_first(col('predError')))

• A. Column predError should be sorted in a descending way, putting nulls last.
• B. Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
• C. Column predError should be sorted by desc_nulls_first() instead.
• D. Column value should be wrapped by the col() operator.
• E. Instead of orderBy, sort should be used.

Answer: A

Explanation:
Explanation
Correct code block:
transactionsDf.orderBy('value', desc_nulls_last('predError'))
Column predError should be sorted in a descending way, putting nulls last.
Correct! By default, Spark sorts ascending, putting nulls first. So, the inverse sort of the default sort is indeed desc_nulls_last.
Instead of orderBy, sort should be used.
No. DataFrame.sort() orders data per partition, it does not guarantee a global order. This is why orderBy is the more appropriate operator here.
Column value should be wrapped by the col() operator.
Incorrect. DataFrame.sort() accepts both string and Column objects.
Column predError should be sorted by desc_nulls_first() instead.
Wrong. Since Spark's default sort order matches asc_nulls_first(), nulls would have to come last when inverted.
Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
No, this would just sort the DataFrame by the very last column, but would not take information from both columns into account, as noted in the question.
More info: pyspark.sql.DataFrame.orderBy - PySpark 3.1.2 documentation, pyspark.sql.functions.desc_nulls_last - PySpark 3.1.2 documentation, sort() vs orderBy() in Spark | Towards Data Science Static notebook | Dynamic notebook: See test 3

 

NEW QUESTION 30
Which of the following describes a valid concern about partitioning?

• A. A shuffle operation returns 200 partitions if not explicitly set.
• B. No data is exchanged between executors when coalesce() is run.
• C. Decreasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.
• D. Short partition processing times are indicative of low skew.
• E. The coalesce() method should be used to increase the number of partitions.

Answer: A

Explanation:
Explanation
A shuffle operation returns 200 partitions if not explicitly set.
Correct. 200 is the default value for the Spark property spark.sql.shuffle.partitions. This property determines how many partitions Spark uses when shuffling data for joins or aggregations.
The coalesce() method should be used to increase the number of partitions.
Incorrect. The coalesce() method can only be used to decrease the number of partitions.
Decreasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.
No. For narrow transformations, fewer partitions usually result in a longer overall runtime, if more executors are available than partitions.
A narrow transformation does not include a shuffle, so no data need to be exchanged between executors.
Shuffles are expensive and can be a bottleneck for executing Spark workloads.
Narrow transformations, however, are executed on a per-partition basis, blocking one executor per partition.
So, it matters how many executors are available to perform work in parallel relative to the number of partitions. If the number of executors is greater than the number of partitions, then some executors are idle while other process the partitions. On the flip side, if the number of executors is smaller than the number of partitions, the entire operation can only be finished after some executors have processed multiple partitions, one after the other. To minimize the overall runtime, one would want to have the number of partitions equal to the number of executors (but not more).
So, for the scenario at hand, increasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.
No data is exchanged between executors when coalesce() is run.
No. While coalesce() avoids a full shuffle, it may still cause a partial shuffle, resulting in data exchange between executors.
Short partition processing times are indicative of low skew.
Incorrect. Data skew means that data is distributed unevenly over the partitions of a dataset. Low skew therefore means that data is distributed evenly.
Partition processing time, the time that executors take to process partitions, can be indicative of skew if some executors take a long time to process a partition, but others do not. However, a short processing time is not per se indicative a low skew: It may simply be short because the partition is small.
A situation indicative of low skew may be when all executors finish processing their partitions in the same timeframe. High skew may be indicated by some executors taking much longer to finish their partitions than others. But the answer does not make any comparison - so by itself it does not provide enough information to make any assessment about skew.
More info: Spark Repartition & Coalesce - Explained and Performance Tuning - Spark 3.1.2 Documentation

 

NEW QUESTION 31
Which of the following describes the characteristics of accumulators?

• A. Accumulators are used to pass around lookup tables across the cluster.
• B. All accumulators used in a Spark application are listed in the Spark UI.
• C. If an action including an accumulator fails during execution and Spark manages to restart the action and complete it successfully, only the successful attempt will be counted in the accumulator.
• D. Accumulators are immutable.
• E. Accumulators can be instantiated directly via the accumulator(n) method of the pyspark.RDD module.

Answer: C

Explanation:
Explanation
If an action including an accumulator fails during execution and Spark manages to restart the action and complete it successfully, only the successful attempt will be counted in the accumulator.
Correct, when Spark tries to rerun a failed action that includes an accumulator, it will only update the accumulator if the action succeeded.
Accumulators are immutable.
No. Although accumulators behave like write-only variables towards the executors and can only be read by the driver, they are not immutable.
All accumulators used in a Spark application are listed in the Spark UI.
Incorrect. For scala, only named, but not unnamed, accumulators are listed in the Spark UI. For pySpark, no accumulators are listed in the Spark UI - this feature is not yet implemented.
Accumulators are used to pass around lookup tables across the cluster.
Wrong - this is what broadcast variables do.
Accumulators can be instantiated directly via the accumulator(n) method of the pyspark.RDD module.
Wrong, accumulators are instantiated via the accumulator(n) method of the sparkContext, for example: counter
= spark.sparkContext.accumulator(0).
More info: python - In Spark, RDDs are immutable, then how Accumulators are implemented? - Stack Overflow, apache spark - When are accumulators truly reliable? - Stack Overflow, Spark - The Definitive Guide, Chapter 14

 

NEW QUESTION 32
Which of the following is one of the big performance advantages that Spark has over Hadoop?

• A. Spark achieves higher resiliency for queries since, different from Hadoop, it can be deployed on Kubernetes.
• B. Spark achieves great performance by storing data in the HDFS format, whereas Hadoop can only use parquet files.
• C. Spark achieves great performance by storing data and performing computation in memory, whereas large jobs in Hadoop require a large amount of relatively slow disk I/O operations.
• D. Spark achieves great performance by storing data in the DAG format, whereas Hadoop can only use parquet files.
• E. Spark achieves performance gains for developers by extending Hadoop's DataFrames with a user-friendly API.

Answer: C

Explanation:
Explanation
Spark achieves great performance by storing data in the DAG format, whereas Hadoop can only use parquet files.
Wrong, there is no "DAG format". DAG stands for "directed acyclic graph". The DAG is a means of representing computational steps in Spark. However, it is true that Hadoop does not use a DAG.
The introduction of the DAG in Spark was a result of the limitation of Hadoop's map reduce framework in which data had to be written to and read from disk continuously.
Graph DAG in Apache Spark - DataFlair
Spark achieves great performance by storing data in the HDFS format, whereas Hadoop can only use parquet files.
No. Spark can certainly store data in HDFS (as well as other formats), but this is not a key performance advantage over Hadoop. Hadoop can use multiple file formats, not only parquet.
Spark achieves higher resiliency for queries since, different from Hadoop, it can be deployed on Kubernetes.
No, resiliency is not asked for in the question. The question is about performance improvements.
Both Hadoop and Spark can be deployed on Kubernetes.
Spark achieves performance gains for developers by extending Hadoop's DataFrames with a user-friendly API.
No. DataFrames are a concept in Spark, but not in Hadoop.

 

NEW QUESTION 33
The code block displayed below contains an error. The code block should return a copy of DataFrame transactionsDf where the name of column transactionId has been changed to transactionNumber. Find the error.
Code block:
transactionsDf.withColumn("transactionNumber", "transactionId")

• A. The method withColumn should be replaced by method withColumnRenamed and the arguments to the method need to be reordered.
• B. Each column name needs to be wrapped in the col() method and method withColumn should be replaced by method withColumnRenamed.
• C. The arguments to the withColumn method need to be reordered and the copy() operator should be appended to the code block to ensure a copy is returned.
• D. The copy() operator should be appended to the code block to ensure a copy is returned.
• E. The arguments to the withColumn method need to be reordered.

Answer: A

Explanation:
Explanation
Correct code block:
transactionsDf.withColumnRenamed("transactionId", "transactionNumber")
Note that in Spark, a copy is returned by default. So, there is no need to append copy() to the code block.
More info: pyspark.sql.DataFrame.withColumnRenamed - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2

 

NEW QUESTION 34
Which of the following describes characteristics of the Spark UI?

• A. Via the Spark UI, stage execution speed can be modified.
• B. The Scheduler tab shows how jobs that are run in parallel by multiple users are distributed across the cluster.
• C. Via the Spark UI, workloads can be manually distributed across executors.
• D. There is a place in the Spark UI that shows the property spark.executor.memory.
• E. Some of the tabs in the Spark UI are named Jobs, Stages, Storage, DAGs, Executors, and SQL.

Answer: D

Explanation:
Explanation
There is a place in the Spark UI that shows the property spark.executor.memory.
Correct, you can see Spark properties such as spark.executor.memory in the Environment tab.
Some of the tabs in the Spark UI are named Jobs, Stages, Storage, DAGs, Executors, and SQL.
Wrong - Jobs, Stages, Storage, Executors, and SQL are all tabs in the Spark UI. DAGs can be inspected in the
"Jobs" tab in the job details or in the Stages or SQL tab, but are not a separate tab.
Via the Spark UI, workloads can be manually distributed across distributors.
No, the Spark UI is meant for inspecting the inner workings of Spark which ultimately helps understand, debug, and optimize Spark transactions.
Via the Spark UI, stage execution speed can be modified.
No, see above.
The Scheduler tab shows how jobs that are run in parallel by multiple users are distributed across the cluster.
No, there is no Scheduler tab.

 

NEW QUESTION 35
......

The Exam cost of Databricks Associate Developer Apache Spark Exam?

The cost of the Databricks Associate Developer Apache Spark Exam is 200 USD per attempt.

How to Register for the Databricks Associate-Developer-Apache-Spark Exam

• You can register for the exam by clicking the Register button.

• Go to create an account.

• The on-screen steps will show you how to arrange an exam with our partner.

• You can see all the available certificate exams by Clicking on the Certifications tab.

 

Associate-Developer-Apache-Spark Dumps Updated Practice Test and 179 unique questions: https://www.test4sure.com/Associate-Developer-Apache-Spark-pass4sure-vce.html

Latest 100% Exam Passing Ratio - Associate-Developer-Apache-Spark Dumps PDF: https://drive.google.com/open?id=1eyjDqbDjZcXsaFhj4fOsmiV_BKQiBvxN