Cuopt Routing API Python

evals/evals.json

[
  {
    "id": "rt-py-eval-001-vrptw-api-call-sequence",
    "question": "For a VRP with time windows in cuopt (Python), list the API calls I need in order — name each method on routing.DataModel and routing.Solve, and one-line what each does. Don't write a full runnable script.",
    "expected_skill": "cuopt-routing-api-python",
    "expected_script": null,
    "ground_truth": "The agent produces an ordered list of API calls without writing executable code. The list, in order: (1) Construct routing.DataModel(n_locations, n_fleet, n_orders). (2) add_cost_matrix(cost_matrix) — pass as a cudf.DataFrame with float32 dtype. (3) add_transit_time_matrix(transit_time_matrix) — required when time windows are used; omitting it causes Solve to return a non-zero status. (4) set_order_locations(series) — cudf.Series of int32 node indices. (5) set_order_time_windows(earliest, latest) — two int32 cudf.Series. (6) Construct routing.SolverSettings(); call set_time_limit() and optionally set_verbose_mode(). (7) Call routing.Solve(dm, ss) to get a solution object. (8) Check solution.get_status() == 0 before reading the route; on a non-zero status, inspect solution.get_error_message() and solution.get_infeasible_orders().to_list(). (9) On success, retrieve the route via solution.get_route() or display it via solution.display_routes(). The agent mentions explicit dtypes (float32 for the matrices, int32 for index series) as a class-level note. Does not embed full executable code, does not invent method names that aren't in the skill (e.g. no fictitious set_time_windows or add_vehicle), and flags that the user must supply real numeric data.",
    "expected_behavior": [
      "Lists the API methods in order without producing a full executable script",
      "Names routing.DataModel with n_locations / n_fleet / n_orders",
      "Names add_cost_matrix and add_transit_time_matrix, and flags that transit_time_matrix is required for time windows",
      "Names set_order_locations and set_order_time_windows",
      "Names routing.SolverSettings (and set_time_limit) and routing.Solve",
      "Mentions checking solution.get_status() == 0, and get_error_message / get_infeasible_orders for the failure path",
      "Mentions explicit dtypes (float32 for matrices, int32 for index series)",
      "Does not invent method names that are not in the skill"
    ]
  },
  {
    "id": "rt-py-eval-002-status-codes-and-triage",
    "question": "routing.Solve came back and my route is empty. How do I find out what happened? What do the status values mean?",
    "expected_skill": "cuopt-routing-api-python",
    "expected_script": null,
    "ground_truth": "The agent explains that solution.get_status() must be checked first, and gives the meaning of the values: 0 = SUCCESS, 1 = FAIL, 2 = TIMEOUT, 3 = EMPTY. For any non-zero status it directs the user to solution.get_error_message() for the reason and solution.get_infeasible_orders().to_list() to see which specific orders could not be served. It also names the usual causes from the skill's issue table — an empty solution generally means time windows are too tight or the travel times are wrong, and infeasible orders generally mean insufficient fleet or capacity. It does not invent additional status codes.",
    "expected_behavior": [
      "States that solution.get_status() == 0 means success and enumerates 1 = FAIL, 2 = TIMEOUT, 3 = EMPTY",
      "Directs the user to solution.get_error_message() and solution.get_infeasible_orders().to_list()",
      "Links an empty solution to over-tight time windows or wrong travel times, and infeasible orders to insufficient fleet or capacity",
      "Does not invent status codes beyond those documented"
    ]
  },
  {
    "id": "rt-py-eval-003-capacity-dimension",
    "question": "My vehicles have a weight limit and each order has a weight. How do I express that in the cuOpt Python routing API?",
    "expected_skill": "cuopt-routing-api-python",
    "expected_script": null,
    "ground_truth": "The agent uses dm.add_capacity_dimension(name, demand_series, capacity_series) — a named dimension carrying the per-order demand and the per-vehicle capacity. It notes that the method is called once per capacity dimension, so multiple dimensions (e.g. weight and volume) mean multiple calls, and that the demand and capacity series should be int32. It does not invent a set_capacity or add_vehicle_capacity method.",
    "expected_behavior": [
      "Names add_capacity_dimension with a dimension name, a per-order demand series, and a per-vehicle capacity series",
      "States that additional capacity dimensions require additional calls",
      "Mentions int32 dtype for the demand/capacity series",
      "Does not invent an alternative capacity method name"
    ]
  },
  {
    "id": "rt-py-eval-004-pickup-delivery-pairs",
    "question": "I need each pickup to be served by the same vehicle that later makes the matching delivery. Which cuOpt Python routing call sets that up?",
    "expected_skill": "cuopt-routing-api-python",
    "expected_script": null,
    "ground_truth": "The agent names dm.set_pickup_delivery_pairs(pickup_indices, delivery_indices), which takes parallel index series pairing each pickup order with its delivery order and makes the problem a PDP. It notes the indices refer to order indices, that the pairing is what enforces same-vehicle and pickup-before-delivery behaviour, and that dm.add_order_precedence(node_id=..., preceding_nodes=...) is the separate mechanism for general precedence between nodes rather than pickup/delivery pairing.",
    "expected_behavior": [
      "Names set_pickup_delivery_pairs with paired pickup and delivery index series",
      "Explains that the pairing is what ties a pickup and its delivery to the same vehicle in order",
      "Distinguishes it from add_order_precedence, which expresses general precedence"
    ]
  },
  {
    "id": "rt-py-eval-005-explicit-dtypes",
    "question": "Do I need to care about dtypes when I build the cost matrix and order location series for cuOpt routing in Python, or will pandas/cudf defaults be fine?",
    "expected_skill": "cuopt-routing-api-python",
    "expected_script": null,
    "ground_truth": "The agent says dtypes must be set explicitly rather than left to defaults: the cost and transit time matrices should be float32, and index-like series such as order locations and demand should be int32. It explains that relying on defaults (e.g. float64 or int64) can produce silent errors rather than a clear exception, which is why the skill calls for explicit casts such as cost_matrix.astype('float32') and cudf.Series([...], dtype='int32').",
    "expected_behavior": [
      "States that float32 is expected for the cost and transit time matrices",
      "States that int32 is expected for order location and demand series",
      "Explains that leaving dtypes to default risks silent errors, so explicit casting is required"
    ]
  },
  {
    "id": "rt-py-eval-006-waypoint-sequence-mutates-route-df",
    "question": "I'm looping over trucks and calling compute_waypoint_sequence on the route dataframe for each one, but after the first truck my cost-matrix indices are wrong. What's going on?",
    "expected_skill": "cuopt-routing-api-python",
    "expected_script": null,
    "ground_truth": "The agent identifies the in-place mutation: compute_waypoint_sequence replaces the location column of the route dataframe with waypoint ids in place, so the original cost-matrix indices are destroyed after the first call and every later iteration reads waypoint ids instead. The fix is to pass route_df.copy() into the call whenever the cost-matrix indices are still needed afterwards, which is exactly the per-truck loop case.",
    "expected_behavior": [
      "Identifies that compute_waypoint_sequence overwrites the location column with waypoint ids in place",
      "Explains that this is why later loop iterations see wrong indices",
      "Gives passing route_df.copy() as the fix"
    ]
  },
  {
    "id": "rt-py-eval-007-no-c-api-for-routing",
    "question": "I'm writing a C application and want to call cuOpt's VRP solver directly from C. How do I do that?",
    "expected_skill": "cuopt-routing-api-python",
    "expected_script": null,
    "ground_truth": "The agent states that routing has no C API in cuOpt — the routing solver is exposed through the Python API only. It redirects the user to a workable alternative: drive routing from Python, or call the cuOpt REST server over HTTP from the C application, since the server is language-agnostic. It does not fabricate C routing entry points or header names.",
    "expected_behavior": [
      "States clearly that cuOpt routing has no C API and is Python-only",
      "Offers the REST server as the language-agnostic route for a non-Python caller",
      "Does not invent C function or header names for routing"
    ]
  }
]